GSoC: Nonparametric methods

[George Panterov]

Dear All,

My name is George Panterov and I am a 5-th year Econ PhD student at
American University. I am one of Dr. Isaac’s students and I am also
Skipper’s classmate. I would like to contribute to the statsmodels
project this summer and I am in the process of writing my GSoC
application.

My primary area of research is financial market microstructure and I
am applying nonparametric estimators to study the effects of
microstructure variables on the moments of return distributions of
assets.

If possible, I would like to contribute to the nonparametric methods (
https://github.com/statsmodels/statsmodels/tree/master/statsmodels/nonparametric
https://github.com/statsmodels/statsmodels/tree/master/statsmodels/sandbox/nonparametric
)
by extending the existing capabilities in the following ways:


1) Expand the number of available kernels by adding the Aitchison and
Aitken’s, Epanechnikov and Wang and Ryzin’s kernels to handle
continuous and discrete (ordered and unordered) random variables and
also work on higher order kernels.
2) Develop the generalized product kernel estimator, which would make
possible to estimate non-parametric conditional and joint densities.
3) Develop the nonparametric kernel regression estimator for models
y=g(x) +e
4) Extend the bandwidth selection procedures to include completely
data driven methods such as likelihood cross-validation and least-
squares cross-validation

I believe that as we get better at collecting data and as our
computational resources continue to increase, nonparametric methods
(albeit more data intensive) will become more and more appalling to
researchers.
I do most of my computational work in python but for nonparametric
estimation I am currently using the package “np” in R and I am
accessing it through rpy2 and rpy which has some drawbacks. This is
why I think it would be great if we could have a python library that
could implement these models.

Please let me know if you have any suggestions and if you think that
this would fit into the statsmodels project. I will post a complete
draft of my GSoC proposal by early next week, but I would appreciate
any input you may have in the meantime.

Thanks,
George

[josef...@gmail.com]

I hope it's appealing :)

> researchers.
> I do most of my computational work in python but for nonparametric
> estimation I am currently using the package “np” in R and I am
> accessing it through rpy2 and rpy which has some drawbacks. This is
> why I think it would be great if we could have a python library that
> could implement these models.
>
> Please let me know if you have any suggestions and if you think that
> this would fit into the statsmodels project. I will post a complete
> draft of my GSoC proposal  by early next week, but I would appreciate
> any input you may have in the meantime.

Hi George,

We want to expand in this area and it is a good topic. Skipper and I
were looking at various papers by Jeffrey Racine for our short to
medium term wishlist, and it would be a desired contribution.

The only possible problem I see with the proposal is that it's not
large enough for a GSoC. Given the work by Skipper and Mike (on kernel
regression), the enhancements might not fill up the time.
I don't think it's a problem with the topic, since nonparametrics is
pretty big, but it might require some thought and planning in the
proposal about the amount of time various parts might take.

If you have some useful methods coming out of the micro-structure
analysis, then this might also make an interesting extra part.

Thanks,

Josef

>
> Thanks,
> George

[George Panterov]

>
> The only possible problem I see with the proposal is that it's not
> large enough for a GSoC. Given the work by Skipper and Mike (on kernel
> regression), the enhancements might not fill up the time.
> I don't think it's a problem with the topic, since nonparametrics is
> pretty big, but it might require some thought and planning in the
> proposal about the amount of time various parts might take.
>
> If you have some useful methods coming out of the micro-structure
> analysis, then this might also make an interesting extra part.
>
> Thanks,
>
> Josef

Hi Josef,

Thanks for the input. I will try to make the proposal as specific as
possible and I will think about some models from the micro-structure
literature.

Since nonparametric methods are computationally intensive a lot of
effort goes into making the code parallel (e.g. npRmpi package in R).
Do you think it would be an interesting addition If I include
something about parallel computation in the proposal or does this go
beyond the scope of the statsmodels project?

[josef...@gmail.com]

A few months ago we added an optional joblib dependency with some
wrapper code. It's what scikit-learn uses for parallel processing. I
think we don't use it much yet, but that would be the easiest
solution.
It will be interesting for statsmodels if we gain more experience with
it or other parallel tools, so we will be able to use it more often.

I don't think we want to go into parallel tool development itself,
since we can leave it to ipython, Gael and the "experts", but we
should make it easy to use it, and use it ourselves as an option for
time consuming code.

Josef

[Skipper Seabold]

To add briefly to this. I think having parallelization for
nonparametric methods is definitely worth some time as part of the
project. That said, with the use of joblib it's almost trivial to
handle embarrassingly parallel problems. It becomes only one line of
code. See, for example,

http://packages.python.org/joblib/parallel.html

And my first stab at cross validation for univariate density estimation

https://github.com/jseabold/statsmodels/blob/kde-crossval/scikits/statsmodels/nonparametric/kde.py#L511

If we do need to incorporate message passing, then more thought would
be needed, but we'd still leverage existing tools. I don't think it
would be necessary with this code though.

Skipper

[George Panterov]

Dear All,

I am posting a more detailed outline of my project proposal for
extending the nonparametric capabilities of statsmodels. I think the
most difficult part is strike the balance between making the project
interesting to GSoC and not undertaking more work than is feasible to
complete for the summer. So if you could provide your thoughts on this
matter it will be greatly appreciated.

The main reference for the models that follow is the text
Nonparametric Econometrics by Qi Li and Jeff Racine but a detailed
reference section will be coming soon.

I would like to extend the nonparametric capabilities of statsmodels
in the following areas:

*Nonparametric Density Estimators*

- Develop a class of density estimators that is able to estimate and
plot unconditional probability densities (PDFs) of the form
P(x_1,x_2,x_3,…,x_n) and cumulative probability densities (CDFs) of
the form P(x_1<X_1,…,x_n<X_n) for mixed data types

- Develop a class of density estimators that is able to estimate and
plot conditional PDFs of the form P(x_1,x_2,…,x_n|y_1,y_2,…,y_n) and
similarly conditional CDFs (based on the generalized product kernel
estimator)

*Nonparametric Models*

- Develop a class that fits nonparametric regression models of the
type y=g(x)+e and implements the local constant kernel estimator and
the local linear kernel estimator proposed by Stone(1977) and
Cleveland (1979) with appropriate significance tests and marginal
effects
- Develop a class that fits nonparametric quantile regression
- Fit nonparametric simultaneous equation models
- Fit nonparametric panel data models


*Semiparametric Models*

- Fit semiparametric partially linear models of the type y=X’b+g(Z)+u
- Fit semiparametric single index models of the type y=g(X’b)+u
- Fit semiparametric Tobit models
- Fit semiparametric Censored Regression Models
- Fit semiparametric partially linear time series model of the type
Y_t=X_t’b + g(Z_t) +u_t

*Bandwidth Selection Procedures*

- The key to good nonparametric estimation is the bandwidth selection.
Please correct me if I am wrong, but statsmodels currently has only
several “rules of thumb” methods for bandwidth selection (like Scott’s
and Silverman’s – bandwidths.py) and not the completely data driven
methods like likelihood cross validation and least-squares cross
validation? I believe that these automatic methods are by far the most
interesting and important part of the nonparametric econometrics
because they are completely data driven which is in tune with the
overall motivation behind using the nonparametric approach in the
first place. The plug-in and “rule-of-thumb” methods like normal
reference rule of thumb, on the other hand, require a specification of
a parametric family of distributions and work well only if your guess
was “close enough”. I am planning to introduce these automatic
methods.

This may be more time-consuming than it appears at first sight. For
example the least-squares cross validation methods involve minimizing
the Integrated Mean Square Error and I suspect that in many cases the
integrals will have difficulties converging so a great deal of
attention and time should be invested in the computational part.
Furthermore, I believe that some of the regression models could be
improved by modifying the bandwidth selection procedures so it may not
be appropriate to use one shoe fits all selection procedure for each
model. If this proves to be the case, then I might have to code a
separate bandwidth selection method for some models.

- I would like to introduce other bandwidth selection methods such as
the one proposed by Hurvich, Simonoff and Tsai (1998) based on the
Akaike information criterion.

- An interesting addition to bandwidth selection procedures could be
an option for breaking down a large sample into smaller components and
averaging over each subsample to obtain the bandwidth. This will make
the estimation more efficient for least-squares cross validation
because the algorithm’s computational time is proportional to the
square of the number of observations so doubling the sample size
increases the run time by a factor of four.

*Kernel Library*

- I don’t think that statsmodels currently has kernels that can handle
discrete ordered an unordered variables. Also I don’t think we could
work with higher-order kernels (please let me know if this isn’t the
case). I would like to develop this further so statsmodels can handle
mixed data estimation.

*Copulas*

- Estimate nonparametric copulas (this is a huge topic but perhaps
some basic capabilities could be introduced)

*Documentation and Examples*

- Introduce examples from economics, finance and financial market
microstructure (possibly recreate some of the examples given by Li and
Racine and implemented in R’s np package)


I suspect that a thorough coverage (including testing) of each of the
bullets above could take about a week to complete and possibly more if
there are some unexpected computational difficulties (such as
convergence problems with bandwidth selection methods).

Do you feel that this general outline would fit within the GSoC
requirements and within the statsmodels goals? Is it too ambitious or
still too short? Should I eliminate and/or expand certain sections?

Thanks for the input

George

[josef...@gmail.com]

On Mon, Mar 26, 2012 at 7:39 PM, George Panterov
<econgp...@gmail.com> wrote:
>
> Dear All,
>
> I am posting a more detailed outline of my project proposal for
> extending the nonparametric capabilities of statsmodels. I think the
> most difficult part is strike the balance between making the project
> interesting to GSoC and not undertaking more work than is feasible to
> complete for the summer. So if you could provide your thoughts on this
> matter it will be greatly appreciated.

I think now it's too much or too ambitious for a GSoC.
But I like it as a future roadmap for possible extensions. Especially,
if you have references that show where to get started.

The outline by topic is also useful, but the actual sequence of coding
would have to be different, I assume. So I would suggest and
additional "time line" that gives the sequence in which the parts are
written. (I don't expect that it will be possible to guess very well
how long the different parts will take)

Also you need to allocate time for writing tests. I would recommend
writing the tests for each part first or along the way, especially if
you are already familiar with the np package. And this test driven
development makes coding much easier.

>
> The main reference for the models that follow is the text
> Nonparametric Econometrics by Qi Li and Jeff Racine but a detailed
> reference section will be coming soon.
>
> I would like to extend the nonparametric capabilities of statsmodels
> in the following areas:
>
> *Nonparametric Density Estimators*
>
> -       Develop a class of density estimators that is able to estimate and
> plot unconditional probability densities (PDFs) of the form
> P(x_1,x_2,x_3,…,x_n) and cumulative probability densities (CDFs) of
> the form P(x_1<X_1,…,x_n<X_n) for mixed data types
>
> -       Develop a class of density estimators that is able to estimate and
> plot conditional PDFs of the form P(x_1,x_2,…,x_n|y_1,y_2,…,y_n) and
> similarly conditional CDFs (based on the generalized product kernel
> estimator)
>
> *Nonparametric Models*
>
> -       Develop a class that fits nonparametric regression models of the
> type y=g(x)+e and implements the local constant kernel estimator and

Is x univariate or multivariate?

> the local linear kernel estimator proposed by Stone(1977) and
> Cleveland (1979) with appropriate significance tests and marginal
> effects
> -       Develop a class that fits nonparametric quantile regression

I don't know much about any details

> -       Fit nonparametric simultaneous equation models
> -       Fit nonparametric panel data models

The last two here should go at the very end, and I doubt there will be
enough time for it.
Both require supporting code that is still unclear or missing, and
might require some time. It will be easier to extend once simultaneous
equation and panel data models are available.
If you have references, I'm interested in it since I don't know what
has been done in this area in the literature.

>
>
> *Semiparametric Models*
>
> -       Fit semiparametric partially linear models of the type y=X’b+g(Z)+u
> -       Fit semiparametric single index models of the type y=g(X’b)+u

> -       Fit semiparametric partially linear time series model of the type
> Y_t=X_t’b + g(Z_t) +u_t

I think these are the best candidates for getting good applications out of the

> - Fit semiparametric Tobit models
> - Fit semiparametric Censored Regression Models

overlap with Skipper's Tobit and with survival/duration models?

>
> *Bandwidth Selection Procedures*

should be early on timeline

very early on time line

>
> -       I don’t think that statsmodels currently has kernels that can handle
> discrete ordered an unordered variables. Also I don’t think we could
> work with higher-order kernels (please let me know if this isn’t the
> case). I would like to develop this further so statsmodels can handle
> mixed data estimation.

One additional part might be bootstrap and bootstrap confidence
intervals, since in many cases convergence to the asymptotic
distribution is too slow (to be very useful in small samples), as far
as I remember.

>
> *Copulas*
>
> -       Estimate nonparametric copulas (this is a huge topic but perhaps
> some basic capabilities could be introduced)

I would say only as far as there might be common code.

>
> *Documentation and Examples*
>
> -       Introduce examples from economics, finance and financial market
> microstructure (possibly recreate some of the examples given by Li and
> Racine and implemented in R’s np package)
>
>
> I suspect that a thorough coverage (including testing) of each of the
> bullets above could take about a week to complete and possibly more if
> there are some unexpected computational difficulties (such as
> convergence problems with bandwidth selection methods).
>
> Do you feel that this general outline would fit within the GSoC
> requirements and within the statsmodels goals? Is it too ambitious or
> still too short? Should I eliminate and/or expand certain sections?
>
> Thanks for the input

I just remembered that I have borrowed Li, Racine 3 months ago, so I
will be able to check some details.

Thanks,

Josef

>
> George

[George Panterov]

Joseph and Skipper, thank you for the useful comments. I am posting
another draft of my GSoC application. If you have the time, I would
appreciate any further suggestions you might have about the abstract
and the project schedule. My goal is to polish up some remaining
issues with the application tomorrow (such as some of the wording),
fix the reference section, post a final draft tomorrow evening to the
mailing list and submit the application on Thursday (one day before
the deadline).


Abstract
-------------------

Statsmodels is a pure Python-based statistics and econometrics package
that has drawn significant attention from applied practitioners from
the fields of Finance, Economics and social sciences. Many of the
basic econometric methods have been developed. In addition some
impressive work has been done in developing the time series methods,
VARs, and DSGE models to name a few. This GSoC I intend to develop the
nonparametric capabilities of statsmodels by focusing in particular on
data-driven bandwidth selection procedures, conditional and
unconditional multivariate probability and cumulative density
estimation and implementing popular nonparametric and semiparametric
regression models.

As we get better at collecting data and as our computational resources
continue to increase, nonparametric methods (despite the fact that
they require more data and are more computationally intensive) will
become more and more appealing to researchers. There are several
commercial packages such as Matlab and Mathematica that can currently
handle some nonparametric estimation. In addition some open source
packages like R which have libraries that can handle some
nonparametric estimation. The goal of this project is to develop an
open-source, Python-based alternative to these sources within
statsmodels which would make the package even more appealing to
practitioners and academics and hopefully make Python the primary
choice for computational work.


Project Schedule
--------------------------

Week 1 – 2 (May 21 – June 3)
Start work on the bandwidth selection methods. Add to the current
“rule-of-thumb” methods, fully data-driven methods such as likelihood
cross and least-squares cross validation and the Hurvich, Simonoff and
Tsai (1998) bandwidth selection method. Introduce several “plug-in”
bandwidth selection procedures for some of the more popular
distributions. This should improve the current univariate kernel
density estimation procedures in statsmodels.

Week 2 – 4 (June 4 – June 17)
Begin work on two major classes: multivariate unconditional density
estimator and multivariate conditional density estimators. Adapt the
existing bandwidth selection procedures to handle the multivariate
density estimation. Create two more classes that will estimate the
cumulative densities in the conditional and unconditional case.

Week 4 – 6 (June 18 – July 1)

Develop a class that fits nonparametric regression models of the type

y=g(x)+e, where x is multivariate, and implements the local constant
kernel estimator and the local linear kernel estimator proposed by

Stone(1977) and Cleveland (1979) with appropriate significance tests
and marginal effects

Week 6 – 8 (July 2 – July 15)
Midterm (July 13) . The work between week 1 and week 6 will form the
backbone of the models to come. Code the appropriate tests for the
conditional, unconditional density estimators and the nonparametric
regression. Cross-check results with the nonparametric package “np”
written for R and make sure all computational methods are working
properly.


Week 8 – 10 (July 16 – July 29)
Begin work on extending the model library. Write two classes that can
fit semiparametric Tobit models and semiparametric censored regression
models.

Week 10 – 12 (July 30 – August 12)
Start work on fitting nonparametric simultaneous equation models and
nonparametric panel data models. These should overlap with the current
existing capabilities of statsmodels[?]. Begin work on the
documentation for the models and start writing tests for the
nonparametric models developed in the second half of the summer.
Compare with results with other existing packages.


Week 12 - (August 13 - )
Polish up and improve any remaining issues with the code. Ensure that
any issues with the documentation are complete.

About me
-------------------
I am currently completing my 9-th semester in the economics PhD
program at American University in Washington, DC. I have completed all
my required course work and I am currently doing research for my
dissertation which is focused on the implementation of nonparametric
and information theoretic methods to continuous double auction
financial markets. A substantial part of my work is on applying kernel-
based method to study the dynamics of asset returns conditional on
microstructure variables such as order flow and order book
characteristics. Currently, I have been using R’s nonparametric
package “np” through rpy2 and rpy but this has its drawbacks. I would
like to help develop the nonparametric capabilities of statsmodels and
contribute to the drive to make python the primary choice for
computational work of academics and professionals alike.

References
--------------------

http://statsmodels.sourceforge.net/
https://github.com/statsmodels/statsmodels/tree/master/statsmodels/nonparametric
http://cran.r-project.org/web/packages/np/index.html


Contact info
-------------------
Name: George Panterov
Email: ......@gmail.com
Phone: 20......
Postal Address: 4000 ......

[josef...@gmail.com]

On Tue, Apr 3, 2012 at 9:23 PM, George Panterov <econgp...@gmail.com> wrote:
> Joseph and Skipper, thank you for the useful comments. I am posting
> another draft of my GSoC application. If you have the time, I would
> appreciate any further suggestions you might have about the abstract
> and the project schedule. My goal is to polish up some remaining
> issues with the application tomorrow (such as some of the wording),
> fix the reference section, post a final draft tomorrow evening to the
> mailing list and submit the application on Thursday (one day before
> the deadline).

Whay not submit it, like, NOW, and update the draft on Thursday? :)

From a quick look, the proposal looks good, but I will go through it
in detail tomorrow.

Thanks,

Josef

[Skipper Seabold]

On Tue, Apr 3, 2012 at 9:45 PM, <josef...@gmail.com> wrote:
>
> On Tue, Apr 3, 2012 at 9:23 PM, George Panterov <econgp...@gmail.com> wrote:
> > Joseph and Skipper, thank you for the useful comments. I am posting
> > another draft of my GSoC application. If you have the time, I would
> > appreciate any further suggestions you might have about the abstract
> > and the project schedule. My goal is to polish up some remaining
> > issues with the application tomorrow (such as some of the wording),
> > fix the reference section, post a final draft tomorrow evening to the
> > mailing list and submit the application on Thursday (one day before
> > the deadline).
>
> Whay not submit it, like, NOW, and update the draft on Thursday? :)
>

Yes, I would go ahead and submit to melange now. I am going to be
doing reviews on there, but a few brief comments below. There have
been many warnings from GSoC/PSF that _melange will break_ sometime
during the application period. It's best to have the application on
there because there won't be another chance.

You may want to add to your before GSoC duties to get familiar with
profiling tools for Python (if you aren't already). I imagine some
time (after everything works) will be spent on trying to optimize the
data-driven methods for speed. I don't find my good profiling tutorial
links right now, but remind me. We'll be able to help here.

> > Week 2 – 4 (June 4 – June 17)
> > Begin work on two major classes: multivariate unconditional density
> > estimator and multivariate conditional density estimators. Adapt the
> > existing bandwidth selection procedures to handle the multivariate
> > density estimation. Create two more classes that will estimate the
> > cumulative densities in the conditional and unconditional case.
> >
> > Week 4 – 6 (June 18 – July 1)
> > Develop a class that fits nonparametric regression models of the type
> > y=g(x)+e, where x is multivariate, and implements the local constant
> > kernel estimator and the local linear kernel estimator proposed by
> > Stone(1977) and Cleveland (1979) with appropriate significance tests
> > and marginal effects
> >

FYI, I have found coding marginal effects efficiently and reusably to
be a big time sink.

> > Week 6 – 8 (July 2 – July 15)
> > Midterm (July 13) . The work between week 1 and week 6 will form the
> > backbone of the models to come. Code the appropriate tests for the
> > conditional, unconditional density estimators and the nonparametric
> > regression. Cross-check results with the nonparametric package “np”
> > written for R and make sure all computational methods are working
> > properly.
> >
> >
> > Week 8 – 10 (July 16 – July 29)
> > Begin work on extending the model library. Write two classes that can
> > fit semiparametric Tobit models and semiparametric censored regression
> > models.
> >

Is the idea of the semiparametric Tobit that it's just a Tobit model
with an unknown underlying distribution?

>
> > Week 10 – 12 (July 30 – August 12)
> > Start work on fitting nonparametric simultaneous equation models and
> > nonparametric panel data models. These should overlap with the current
> > existing capabilities of statsmodels[?]. Begin work on the
> > documentation for the models and start writing tests for the
> > nonparametric models developed in the second half of the summer.
> > Compare with results with other existing packages.
> >

This last part is very ambitious. I don't know that we'll make it here
and if we do that 2 weeks will be enough time to cover this. Overall
though the proposal looks good and properly focused.

Skipper

[George Panterov]

My Final Draft before submitting:

Abstract
-------------------

Statsmodels is a pure Python-based statistics and econometrics package
that has drawn significant attention from applied practitioners from
the fields of Finance, Economics and social sciences. Many of the
basic econometric methods have been developed. In addition some
impressive work has been done in developing the time series methods,
VARs, and DSGE models to name a few. This GSoC I intend to develop the
nonparametric capabilities of statsmodels by focusing in particular on
data-driven bandwidth selection procedures, conditional and
unconditional multivariate probability and cumulative density
estimation and implementing popular nonparametric and semiparametric
regression models.

The Project
------------------

As we get better at collecting data and as our computational resources
continue to increase, nonparametric methods (despite the fact that
they require more data and are more computationally intensive) will
become more and more appealing to researchers. There are several
commercial packages such as Matlab and Mathematica that can currently
handle some nonparametric estimation. In addition some open source

packages like R have libraries that can handle some nonparametric
estimation [4]. The goal of this project is to develop an open-source,
Python-based alternative to these sources within statsmodels (see [1]
and [6]) which would make the package even more appealing to

practitioners and academics and hopefully make Python the primary
choice for computational work.

The main focus of my summer work will be to expand the current
nonparametric capabilities of statsmodels [2,3] in three main
directions: develop the fully data-driven bandwidth selection methods
and improve the existing “rule-of-thumb” methods; make it possible to
handle conditional and unconditional multivariate kernel density
estimation; and work on popular nonparametric models (see the textbook
Nonparametric Econometrics by Qi Li and Jeff Racine, 2007)


Project Schedule
--------------------------


Pre-GSoC
Get familiar with the profiling tools for Python and organize and
familiarize with the existing code in the sandbox [3]. Look for
tutorials for optimization for speed of the data-driven methods for
bandwidth selection.

Week 1 – 2 (May 21 – June 3)
Start work on the bandwidth selection methods. Add to the current
“rule-of-thumb” methods, fully data-driven methods such as likelihood
cross and least-squares cross validation and the Hurvich, Simonoff and
Tsai (1998) bandwidth selection method. Introduce several “plug-in”
bandwidth selection procedures for some of the more popular
distributions. This should improve the current univariate kernel
density estimation procedures in statsmodels.

Week 2 – 4 (June 4 – June 17)
Begin work on two major classes: multivariate unconditional density
estimator and multivariate conditional density estimators. Adapt the
existing bandwidth selection procedures to handle the multivariate
density estimation. Create two more classes that will estimate the
cumulative densities in the conditional and unconditional case.

Week 4 – 6 (June 18 – July 1)
Develop a class that fits nonparametric regression models of the type
y=g(x)+e, where x is multivariate, and implements the local constant
kernel estimator and the local linear kernel estimator proposed by
Stone(1977) and Cleveland (1979) with appropriate significance tests
and marginal effects

Week 6 – 8 (July 2 – July 15)
Midterm (July 13) . The work between week 1 and week 6 will form the
backbone of the models to come. Code the appropriate tests for the
conditional, unconditional density estimators and the nonparametric
regression. Cross-check results with the nonparametric package “np”
written for R and make sure all computational methods are working

properly [4].

Week 8 – 10 (July 16 – July 29)
Begin work on extending the model library. Write two classes that can
fit semiparametric Tobit models and semiparametric censored regression
models.

Week 10 – 12 (July 30 – August 12)

Explore the feasibility of including more advanced models such as

nonparametric simultaneous equation models and nonparametric panel

data models. Check if there is existing code that overlaps and start
the groundwork. These should overlap with the current existing
capabilities of statsmodels [1]. Begin work on the documentation for

the models and start writing tests for the nonparametric models
developed in the second half of the summer. Compare with results with
other existing packages.

Week 12 - (August 13 - )
Polish up and improve any remaining issues with the code. Ensure that
any issues with the documentation are complete.


About me
-------------------
I am currently completing my 9-th semester in the economics PhD
program at American University in Washington, DC. I have completed all
my required course work and I am currently doing research for my
dissertation which is focused on the implementation of nonparametric
and information theoretic methods to continuous double auction
financial markets. A substantial part of my work is on applying kernel-

based methods to study the dynamics of asset returns conditional on

microstructure variables such as order flow and order book

characteristics. Some of my Python code on nonparametric estimation,
time series, microstructure and genetic algorithms is publicly
available [5]. Currently, I have been using R’s nonparametric package

“np” through rpy2 and rpy but this has its drawbacks. I would like to
help develop the nonparametric capabilities of statsmodels and

contribute to the drive to make python the primary choice for

computational work of academics and professionals alike.

References
--------------------

[1] http://statsmodels.sourceforge.net/
[2] https://github.com/statsmodels/statsmodels/tree/master/statsmodels/nonparametric
[3] https://github.com/statsmodels/statsmodels/tree/master/statsmodels/sandbox/nonparametric
[4] http://cran.r-project.org/web/packages/np/index.html
[5] https://github.com/gpanterov/Nonparametric-kernel-density-estimation
[6] https://code.launchpad.net/statsmodels

Contact info
-------------------
Name: George Panterov

Project Blog: http://statsmodels-np.blogspot.com/
Project Wiki: https://github.com/statsmodels/statsmodels/wiki/GSoC-Ideas

Email: ......@gmail.com
Phone: 20......
Postal Address: 4000 ......

Github: gpanterov

[Skipper Seabold]

On Thu, Apr 5, 2012 at 1:45 PM, George Panterov <econgp...@gmail.com> wrote:
> My Final Draft before submitting:
>

Go ahead and submit. You can edit until the deadline. They keep
warning that Melange will crash in the next 24 hours and that people
always have problems and don't get to submit but the deadline is still
final.

[George Panterov]

> Go ahead and submit. You can edit until the deadline. They keep
> warning that Melange will crash in the next 24 hours and that people
> always have problems and don't get to submit but the deadline is still
> final.
>

It is submitted. Thanks
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/gpanterov/1

[josef...@gmail.com]

I read the melange version (in case there are differences)

I think in the following parts the word "kernel" should be used once
or twice to emphasise that it is for kernel based non-parametrics, not
splines or orthogonal polynomials for example.

>
> Week 1 – 2 (May 21 – June 3)
> Start work on the bandwidth selection methods. Add to the current
> “rule-of-thumb” methods, fully data-driven methods such as likelihood
> cross and least-squares cross validation and the Hurvich, Simonoff and
> Tsai (1998) bandwidth selection method. Introduce several “plug-in”
> bandwidth selection procedures for some of the more popular
> distributions. This should improve the current univariate kernel
> density estimation procedures in statsmodels.

What happened to the kernels for categorical variables?

>
> Week 2 – 4 (June 4 – June 17)
> Begin work on two major classes: multivariate unconditional density
> estimator and multivariate conditional density estimators. Adapt the
> existing bandwidth selection procedures to handle the multivariate
> density estimation. Create two more classes that will estimate the
> cumulative densities in the conditional and unconditional case.
>
> Week 4 – 6 (June 18 – July 1)
> Develop a class that fits nonparametric regression models of the type
> y=g(x)+e, where x is multivariate, and implements the local constant
> kernel estimator and the local linear kernel estimator proposed by
> Stone(1977) and Cleveland (1979) with appropriate significance tests
> and marginal effects

statsmodels has lowess which is based on one of the Cleveland papers,
but without statistical results, like confidence intervals

maybe y=g(x) + Z beta + e as extension ?

>
> Week 6 – 8 (July 2 – July 15)
> Midterm (July 13) . The work between week 1 and week 6 will form the
> backbone of the models to come. Code the appropriate tests for the
> conditional, unconditional density estimators and the nonparametric
> regression. Cross-check results with the nonparametric package “np”
> written for R and make sure all computational methods are working
> properly [4].
>
>
> Week 8 – 10 (July 16 – July 29)
> Begin work on extending the model library. Write two classes that can
> fit semiparametric Tobit models and semiparametric censored regression
> models.
>
> Week 10 – 12 (July 30 – August 12)
> Explore the feasibility of including more advanced models such as
> nonparametric simultaneous equation models and nonparametric panel
> data models. Check if there is existing code that overlaps and start
> the groundwork. These should overlap with the current existing
> capabilities of statsmodels [1]. Begin work on the documentation for
> the models and start writing tests for the nonparametric models
> developed in the second half of the summer. Compare with results with
> other existing packages.

Instead of nonparametric simultaneous equations (which might not be
easy) applying the kernel methods to statistical tests might be more
interesting.
e.g non-parametric specification tests: compare/test of a parametric
model against a kernel-based non-parametric alternative.

Overall it looks good,
I also recommend test and document as you go, and consider the time at
the end more for cleanup and filling some holes in documentation and
tests.

Josef

[George Panterov]

Josef, thanks for the suggestion. I will add the categorical kernel
density estimation ( I don't know how I dropped it. I was always
planning to do it)

In terms of the testing and documentation. I also agree that testing
and documenting as you code is the best practice. In fact, I was
planning to do it that way and then have a few weeks around the
midterm and final where I would just spend some more time on it.
However, in terms of writing the application I was hesitant to add to
each point in the timeline "testing and documentation". But if you
think that this would make a stronger application I will consider
reworking it.

On Apr 5, 3:54 pm, josef.p...@gmail.com wrote:

> > [2]https://github.com/statsmodels/statsmodels/tree/master/statsmodels/no...
> > [3]https://github.com/statsmodels/statsmodels/tree/master/statsmodels/sa...

[josef...@gmail.com]

On Thu, Apr 5, 2012 at 4:29 PM, George Panterov <econgp...@gmail.com> wrote:
> Josef, thanks for the suggestion. I will add the categorical kernel
> density estimation ( I don't know how I dropped it. I was always
> planning to do it)
>
> In terms of the testing and documentation. I also agree that testing
> and documenting as you code is the best practice. In fact, I was
> planning to do it that way and then have a few weeks around the
> midterm and final where I would just spend some more time on it.
> However, in terms of writing the application I was hesitant to add to
> each point in the timeline "testing and documentation". But if you
> think that this would make a stronger application I will consider
> reworking it.

No, I don't think it's necessary to add it everywhere. At most mention it once.
I just wanted to clarify that having a section "testing and
documentation" doesn't mean it's done only during that time.

Josef

[George Panterov]

Hi All,
I am working on my code submission for the PSF requirements and I have
a quick question about the API for the nonparametric methods.

So I am looking at the kernels.py file and the CustomKernel class has
a density method that is supposed to return the density. However we
also have kernel density estimators in kde.py. Isn't this redundant?
Wouldn't it be better if our kernel library consist only of kernel
functions which take two inputs and return a single value i.e --

value=K((X_i - x)/h)

and then perform the kernel density estimation separately --

f(x)=(1/nh)*SUM_i [K((X_i - x)/h)].


I think this would make it much easier to transition from univariate
to multivariate density estimation with the "generalized product
kernels" method.

(I also keep getting errors when I try to work with the
CustomKernel.density() method... )

Of course it is possible that I am missing/misreading something...
Please let me know if this is the case.

[josef...@gmail.com]

On Sat, Apr 14, 2012 at 8:53 PM, George Panterov
<econgp...@gmail.com> wrote:
> Hi All,
> I am working on my code submission for the PSF requirements and I have
> a quick question about the API for the nonparametric methods.
>
> So I am looking at the kernels.py file and the CustomKernel class has
> a density method that is supposed to return the density. However we
> also have kernel density estimators in kde.py. Isn't this redundant?

I think the statsmodels.nonparametric.kde is a partially rewritten
version by Skipper, but as far as I know Skipper then focused on
binned kernel density estimation with fft.
sandbox.nonparametric.kernels is a class that is used and has helper
methods for kernel density and kernel regression/smoothers. The main
KernelSmoother class is the one in sandbox.nonparametric.smoothers, I
think.

How these classes will look like at the end is still a bit open, I guess.
Both KDE, and KernelSmoother use methods from CustomKernel and it's
subclasses under the hood.

> Wouldn't it be better if our kernel library consist only of kernel
> functions which take two inputs and return a single value  i.e --
>
> value=K((X_i - x)/h)
>
> and then perform the kernel density estimation separately --
>
> f(x)=(1/nh)*SUM_i [K((X_i - x)/h)].

>
>
> I think this would make it much easier to transition from univariate
> to multivariate density estimation with the "generalized product
> kernels" method.

I think we should somewhere have a "pure" kernel window module,
similar to the windows in numpy or scipy. But I don't think it will
make the additional methods that CustomKernel has obsolete, since they
are useful. There are usages for kernel windows in tsa, and for GMM.
The advantage of sticking kernels into a class is that we can add
other things that we might need, domain, derivative, integral (cdf),
... (I don't know what we might need).

On the other hand, when we go more to cython for the expensive code in
this area, we might just want to duplicate the kernel windows in
cython.

>
> (I also keep getting errors when I try to work with the
> CustomKernel.density() method... )
>
> Of course it is possible that I am missing/misreading something...
> Please let me know if this is the case.

It's possible that there are refactoring victims, I cannot find an
example script right now, and I don't see any unit tests.

Thanks,

Josef

[Ralf Gommers]

On Sun, Apr 15, 2012 at 3:51 AM, <josef...@gmail.com> wrote:

On Sat, Apr 14, 2012 at 8:53 PM, George Panterov
<econgp...@gmail.com> wrote:
> Hi All,
> I am working on my code submission for the PSF requirements and I have
> a quick question about the API for the nonparametric methods.
>
> So I am looking at the kernels.py file and the CustomKernel class has
> a density method that is supposed to return the density. However we
> also have kernel density estimators in kde.py. Isn't this redundant?

I think the statsmodels.nonparametric.kde is a partially rewritten
version by Skipper, but as far as I know Skipper then focused on
binned kernel density estimation with fft.
sandbox.nonparametric.kernels is a class that is used and has helper
methods for kernel density and kernel regression/smoothers. The main
KernelSmoother class is the one in sandbox.nonparametric.smoothers, I
think.

That density method does look a bit redundant.

Is everything in sandbox considered unstable (so modifiable whenever that would be convenient)? sandbox.nonparametric.kernels is used by nonparametric.kde, but I assume that if kde keeps working as advertised sandbox code can be changed?
 

How these classes will look like at the end is still a bit open, I guess.
Both KDE, and KernelSmoother use methods from CustomKernel and it's
subclasses under the hood.

> Wouldn't it be better if our kernel library consist only of kernel
> functions which take two inputs and return a single value  i.e --
>
> value=K((X_i - x)/h)
>
> and then perform the kernel density estimation separately --
>
> f(x)=(1/nh)*SUM_i [K((X_i - x)/h)].

>
>
> I think this would make it much easier to transition from univariate
> to multivariate density estimation with the "generalized product
> kernels" method.

I haven't read the original Li and Racine 2003 paper on this in too much detail yet, but the basic idea is to simply calculate simple product kernels for continuous and discrete variables separately, then multiply those? At first sight I don't see anything that would cause you to make different choices for your software design here with respect to the case of only having product kernels of continuous variables.

Ralf

[Skipper Seabold]

On Mon, Apr 16, 2012 at 4:34 PM, Ralf Gommers
<ralf.g...@googlemail.com> wrote:
>
>
>
> On Sun, Apr 15, 2012 at 3:51 AM, <josef...@gmail.com> wrote:
>>
>> On Sat, Apr 14, 2012 at 8:53 PM, George Panterov
>> <econgp...@gmail.com> wrote:
>> > Hi All,
>> > I am working on my code submission for the PSF requirements and I have
>> > a quick question about the API for the nonparametric methods.
>> >
>> > So I am looking at the kernels.py file and the CustomKernel class has
>> > a density method that is supposed to return the density. However we
>> > also have kernel density estimators in kde.py. Isn't this redundant?
>>
>> I think the statsmodels.nonparametric.kde is a partially rewritten
>> version by Skipper, but as far as I know Skipper then focused on
>> binned kernel density estimation with fft.
>> sandbox.nonparametric.kernels is a class that is used and has helper
>> methods for kernel density and kernel regression/smoothers. The main
>> KernelSmoother class is the one in sandbox.nonparametric.smoothers, I
>> think.
>
>
> That density method does look a bit redundant.

Sorry I need to catch up on this thread. IIRC that density method is a
naive density estimator ie., it does the sums over a grid. My version
uses fft for the gaussian kernel.

>
> Is everything in sandbox considered unstable (so modifiable whenever that would be convenient)? sandbox.nonparametric.kernels is used by nonparametric.kde, but I assume that if kde keeps working as advertised sandbox code can be changed?
>

I think this is fair. My impression from the sandbox kernels was that
they could be made to benefit from a refactor. I'll have to
refamiliarize myself with this code.

[Ralf Gommers]

Do you know where they came from? Git isn't helpful with the history after the scikits-->statsmodels rename.

Some improvements can certainly be made to those kernels, but it would be helpful to understand why all those methods are there, and which ones we want to keep.

Ralf

[josef...@gmail.com]

On Mon, Apr 16, 2012 at 4:34 PM, Ralf Gommers
<ralf.g...@googlemail.com> wrote:
>
>

> On Sun, Apr 15, 2012 at 3:51 AM, <josef...@gmail.com> wrote:
>>
>> On Sat, Apr 14, 2012 at 8:53 PM, George Panterov
>> <econgp...@gmail.com> wrote:
>> > Hi All,
>> > I am working on my code submission for the PSF requirements and I have
>> > a quick question about the API for the nonparametric methods.
>> >
>> > So I am looking at the kernels.py file and the CustomKernel class has
>> > a density method that is supposed to return the density. However we
>> > also have kernel density estimators in kde.py. Isn't this redundant?
>>
>> I think the statsmodels.nonparametric.kde is a partially rewritten
>> version by Skipper, but as far as I know Skipper then focused on
>> binned kernel density estimation with fft.
>> sandbox.nonparametric.kernels is a class that is used and has helper
>> methods for kernel density and kernel regression/smoothers. The main
>> KernelSmoother class is the one in sandbox.nonparametric.smoothers, I
>> think.
>
>
> That density method does look a bit redundant.
>
> Is everything in sandbox considered unstable (so modifiable whenever that
> would be convenient)? sandbox.nonparametric.kernels is used by
> nonparametric.kde, but I assume that if kde keeps working as advertised
> sandbox code can be changed?

Essentially yes, there are a few places where sandbox code get's imported
into the main statsmodels, where we might want to be cautious but sandbox
code is mostly in the sandbox because it needs to be refactored.
So, essentially no constraint on improving the code.

>
>>
>> How these classes will look like at the end is still a bit open, I guess.
>> Both KDE, and KernelSmoother use methods from CustomKernel and it's
>> subclasses under the hood.
>>
>>
>> > Wouldn't it be better if our kernel library consist only of kernel
>> > functions which take two inputs and return a single value  i.e --
>> >
>> > value=K((X_i - x)/h)
>> >
>> > and then perform the kernel density estimation separately --
>> >
>> > f(x)=(1/nh)*SUM_i [K((X_i - x)/h)].
>>
>> >
>> >
>> > I think this would make it much easier to transition from univariate
>> > to multivariate density estimation with the "generalized product
>> > kernels" method.
>
>
> I haven't read the original Li and Racine 2003 paper on this in too much
> detail yet, but the basic idea is to simply calculate simple product kernels
> for continuous and discrete variables separately, then multiply those? At
> first sight I don't see anything that would cause you to make different
> choices for your software design here with respect to the case of only
> having product kernels of continuous variables.

On the other hand, it might be better to outsource some code to
functions if it is used by different classes for different purposes.

Josef

[Skipper Seabold]

On Mon, Apr 16, 2012 at 4:49 PM, Ralf Gommers

If you click on history, you can follow it back to the old file in
github. Mike had originally written them for Kernel Regression. My
vague recollection is I took only the parts I needed and did the rest
for small speed gains.

https://github.com/statsmodels/statsmodels/commits/master/scikits/statsmodels/sandbox/nonparametric/kernel.py

[josef...@gmail.com]

starting at line 21
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/sandbox/nonparametric/smoothers.py
KernelSmoother uses the methods that are attached to kernels

one possibility, if it works, would be to separate the pure kernel
(windows) code from the code that uses it, kde or kernel regression,
1d versus nd.

But I also haven't looked at all the details

Josef

[Ralf Gommers]

On Mon, Apr 16, 2012 at 10:51 PM, Skipper Seabold <jsse...@gmail.com> wrote:

On Mon, Apr 16, 2012 at 4:49 PM, Ralf Gommers

<ralf.g...@googlemail.com> wrote:
>
>
> On Mon, Apr 16, 2012 at 10:38 PM, Skipper Seabold <jsse...@gmail.com>
> wrote:
>>
>> On Mon, Apr 16, 2012 at 4:34 PM, Ralf Gommers
>> <ralf.g...@googlemail.com> wrote:
>>
>> >
>> > Is everything in sandbox considered unstable (so modifiable whenever
>> > that would be convenient)? sandbox.nonparametric.kernels is used by
>> > nonparametric.kde, but I assume that if kde keeps working as advertised
>> > sandbox code can be changed?
>> >
>>
>> I think this is fair. My impression from the sandbox kernels was that
>> they could be made to benefit from a refactor. I'll have to
>> refamiliarize myself with this code.
>
>
> Do you know where they came from? Git isn't helpful with the history after
> the scikits-->statsmodels rename.

If you click on history, you can follow it back to the old file in
github.

Cool, Github >> gitk.
 

[Ralf Gommers]

Makes sense. Some attributes like `domain` may still be useful, but methods like `smooth` and `smoothvar` belong in KernelSmoother rather than in CustomKernel I think. There's actually not much left if you remove all the smooth/density methods (and InDomain looks fishy too), so perhaps George's approach makes sense to start with. Would be easy enough to switch from functions to/from class instances later.

Ralf

[josef...@gmail.com]

On Mon, Apr 16, 2012 at 5:21 PM, Ralf Gommers

One problem is where to put kernel specific auxiliary code.
KernelSmoother looks like a generic class that doesn't have kernel
specific info. smooth, smoothvar and similar might have kernel
specific explicit expressions (mostly guessing, only Gaussian has
explicit smooth).

A bit similar to the families and links in genmod.

Josef

[josef...@gmail.com]

browsing the code a bit more:

Biweight also has smooth and smoothvar.

I think we shouldn't kill/strip these classes in kernels until we have
a better idea how kernel regression/smoothing will be designed.

It looks to me that the `shape` lambda functions in the __init__
method are the kernel windows that we want to outsource, or could be
made into methods of the subclasses.
My guess is that many of the list comprehension could be replaced by
numpy array operations.

I think for kernel distribution (cdf not density) estimation it would
also be good to add the integral of the kernel windows as methods.

Josef

[josef...@gmail.com]

digging in the history and trying to find runable examples

statsmodels\sandbox\examples\try_smoothers.py
has examples with plots for the KernelSmoother that uses kernels.py
the second plot has lowess and polynomial fit overlayed for comparison

The example in __main__ of statsmodels\sandbox\nonparametric\kde2.py
runs after updating the imports
2d might not work

Josef

[josef...@gmail.com]

since it's difficult to resist some late fun, a plot

kde2.KDE with 2d example, old faithful

Josef

[Ralf Gommers]

That's a good argument for not putting everything in the generic KernelSmoother class. There are a lot of options here though. For example, kernel functions plus separate smoothing/regression classes could make sense. Using kernels with smooth() methods in regression perhaps less so.

 

I think we shouldn't kill/strip these classes in kernels until we have
a better idea how kernel regression/smoothing will be designed.

True. Having a set of examples like try_smoothers.py to work with helps in forming those ideas.
 

It looks to me that the `shape` lambda functions in the __init__
method are the kernel windows that we want to outsource, or could be
made into methods of the subclasses.

Agreed. Lambda functions in __init__ signatures look really funny anyway.
 

My guess is that many of the list comprehension could be replaced by
numpy array operations.

I think for kernel distribution (cdf not density) estimation it would
also be good to add the integral of the kernel windows as methods.

 That's also why scipy.stats.gaussian_kde has integration methods I suppose.

Ralf

[George Panterov]

Hello again everyone,

I would like to pick your brains on a problem I am working on.

I started some preliminary work on the nonparametric conditional
density estimator -- P(Y1,Y2,...Yn | X1,X2....Xk). The user needs to
input not only a set of dependent vars and a set of independent
variables but also specify the type of the variable (continuous,
discrete ordered or discrete unordered) so that a proper kernel can be
used in the estimation. In R, there is a natural and easy way to do
this. For example:

npcdens(Y1 + ordered(Y2) ~ X1 + factor(X2) + X3, bw)

returns the conditional density P(Y1, Y2 | X1,X2, X3) given a
bandwidth bw. However, this is not so easy to do in Python.

One solution I thought could work is that dep/indep vars and var types
could be attributes of the class Estimator. For example

Estimator.DepVars=[Y1,Y2]
Estimator.IndepVars=[X1,X2,X3]
Estimator.Ordered_Vars=[Y2]
Estimator.Unordered_Vars=[X2]

Estimator.Estimate()

What would be a less cumbersome way for the user to specify dependent/
indep variables and variable types ?

Thanks
George

[josef...@gmail.com]

Can the dependent variables also be ordered or categorical?

When I looked at the nd kde version in the sandbox, I thought you
would need a list of kernels for each dependent variable, if you work
with product kernels?
kde_conditional(endog, exog, kernels=a list of kernels).
kde_mv(endog, kernels=a list of kernels).

another way would be to require an idx for the type of variable as
keyword argument, if the variable is not continuous,
kde_conditional(endog, exog, vartype= [.....]).
kde_conditional(endog, exog, isordered=[1,3]), iscategorical=2).
(separate for exog endog or encoded?)

In general, required information should go into the arguments of the
__init__ of a class, and it should not be required to attach
information after creating an instance.
For now, I would just add keyword arguments with all the additional
information that you need, and we decide on fine-tuning the user
interface later.

for categorical variables, all you get is different kde's for each
level of the variable, is it?
For other models I was thinking of adding a BY keyword similar to SAS
for stratas/subsamples. It's not clear to me yet how to add this, but
Scott has added stratification in survival which might be useful also
with other models.

Josef

>
> Thanks
> George

[George Panterov]

>
> Can the dependent variables also be ordered or categorical?
>

Yes. They could be any of the three types (continuous, ordered,
unordered)

> kde_conditional(endog, exog, vartype= [.....]).
> kde_conditional(endog, exog, isordered=[1,3]), iscategorical=2).

I like this approach. But here again we need to differenciate between
endog and exog since you can have ordered endog variables. Maybe
kde_conditional(endog, exog, isordered_endog=2, isordered_exog=[1,3]),
iscategorical_exog=2).

I am afraid this is maybe too cluttered (compared to R for example)

> for categorical variables, all you get is different kde's for each
> level of the variable, is it?

>
>
>
>
>
>
>
>
>
>

> > Thanks
> > George

[Alan G Isaac]

On 4/21/2012 3:18 PM, George Panterov wrote:
> npcdens(Y1 + ordered(Y2) ~ X1 + factor(X2) + X3, bw)
>
> returns the conditional density P(Y1, Y2 | X1,X2, X3) given a
> bandwidth bw. However, this is not so easy to do in Python.

You mean, with the built in types ...

So that is the basic decisison: to work with builtin types
and NumPy arrays, or to create your own types.

In either case, the information you want to associate with
the variables must either be added at estimation type or
be somehow tagged to the variable.

Here are one possibility.

Estimator((Y1,Y2),(X1,X2,X3),deptypes='co',indtypes='cfc')

(You may prefer to be more verbose/explicit.)

You may also want to explore whether you can adapt
the work on formulae:
http://statsmodels.sourceforge.net/dev/roadmap_todo.html

Alan Isaac

[Ralf Gommers]

On Sun, Apr 22, 2012 at 3:29 PM, Alan G Isaac <ais...@american.edu> wrote:

On 4/21/2012 3:18 PM, George Panterov wrote:

npcdens(Y1 + ordered(Y2) ~ X1 + factor(X2) + X3, bw)

returns the conditional density P(Y1, Y2 | X1,X2, X3) given a
bandwidth bw. However, this is not so easy to do in Python.

You mean, with the built in types ...

So that is the basic decisison: to work with builtin types
and NumPy arrays, or to create your own types.

In either case, the information you want to associate with
the variables must either be added at estimation type or
be somehow tagged to the variable.

Here are one possibility.

Estimator((Y1,Y2),(X1,X2,X3),deptypes='co',indtypes='cfc')

You can also come very close to the R syntax by providing functions "ordered" and "factor" that return a ndarray subclass containing the input array plus one added attribute carrying the required info. It could look like

    npcdens((Y1, ordered(Y2)), (X1, factor(X2), X3), bw)

I think these are details that are easier to fill in once you have something working though, instead of designing everything up-front. Changing from one API to the other will be fairly quick.

Ralf

[Skipper Seabold]

On Sat, Apr 21, 2012 at 5:03 PM, George Panterov
<econgp...@gmail.com> wrote:
>
>>
>> Can the dependent variables also be ordered or categorical?
>>
> Yes. They could be any of the three types (continuous, ordered,
> unordered)
>
>> kde_conditional(endog, exog, vartype= [.....]).
>> kde_conditional(endog, exog, isordered=[1,3]), iscategorical=2).
>
> I like this approach. But here again we need to differenciate between
> endog and exog since you can have ordered endog variables. Maybe
> kde_conditional(endog, exog, isordered_endog=2, isordered_exog=[1,3]),
> iscategorical_exog=2).
>
> I am afraid this is maybe too cluttered (compared to R for example)
>

I'm +1 for the lists of column indices like this.

To avoid the endog/exog, it might make sense to have

kde_conditional_cont
kde_conditional_ordered
kde_conditional_categorical

Where _cont, _ordered, and _categorical are to indicate the nature of
the endogenous variable.

I wouldn't worry too much about this. Soon, I think, users will never
see this, and you'll be able to just pass in "Y1 + ordered(Y2) ~ X1 +
factor(X2) + X3" to kde_conditional and then under the hood it figures
things out and delegates to this more verbose API. The Formula stuff
is coming along. Next on my list is to add support for ordered and
unordered factors to formula.

And we can always rethink the design later when things are nicer. This
will be cheap.

Skipper

[Skipper Seabold]

On Mon, Apr 23, 2012 at 4:53 PM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Sat, Apr 21, 2012 at 5:03 PM, George Panterov
> <econgp...@gmail.com> wrote:
>>
>>>
>>> Can the dependent variables also be ordered or categorical?
>>>
>> Yes. They could be any of the three types (continuous, ordered,
>> unordered)
>>
>>> kde_conditional(endog, exog, vartype= [.....]).
>>> kde_conditional(endog, exog, isordered=[1,3]), iscategorical=2).
>>
>> I like this approach. But here again we need to differenciate between
>> endog and exog since you can have ordered endog variables. Maybe
>> kde_conditional(endog, exog, isordered_endog=2, isordered_exog=[1,3]),
>> iscategorical_exog=2).
>>
>> I am afraid this is maybe too cluttered (compared to R for example)
>>
>
> I'm +1 for the lists of column indices like this.
>
> To avoid the endog/exog, it might make sense to have
>
> kde_conditional_cont
> kde_conditional_ordered
> kde_conditional_categorical
>
> Where _cont, _ordered, and _categorical are to indicate the nature of
> the endogenous variable.
>

Though I already see that this doesn't work on the below example. Maye
we can't avoid the differentiated column indices for now.

[josef...@gmail.com]

And some point we need to discuss more generally the API both for
internal use and user facing for models that require more than a few
keywords, so we can get a roughly consistent structure.
In one-way panel/repeated mixed I need groups indicators, the old
Mixed class still requires Units, Scott in survival uses a Survival
class as endog that holds the timing and censoring information, Tobit
requires extra data, ...

right now I would decide mostly on internal demand, e.g. do ordered
and categorical need to be integers or can they be floats?

string based formulas won't be convenient for internal/programmatic use.

Josef

>>
>> Skipper

[josef...@gmail.com]

https://github.com/gpanterov/statsmodels/commit/4be284ecc7ce01003b0f2c3cfe72627670e7fc60#L0L78

if you do equality checks then it would be best just to work with
integers, and then having a separate array argument would save some
copying

kde_conditional(endog, exog, endog_cat=y2, exog_cat=x2)

where do I specify which kernel I want? What about kernels with a
boundary support?

>>> And we can always rethink the design later when things are nicer. This
>>> will be cheap.

Josef

[George Panterov]

Dear All,

Thanks for the comments on the API. I have been working on the density estimators and I pushed the first version of the code. I know that the code is woefully under-commented and that there are many things to do in terms of style and improve readability. If you don't want to read the current version please wait a day or two until I submit a more cleaned-up version. But I decided to post it anyway so that if there are issues with it I can start fixing them. Also the test file still uses rpy2 ...

The code replicates almost identically the results in R with the normal reference bandwidth selection method. From time to time there are issues with cross validation maximum likelihood method for bandwidth selection because of failure to find the optimal value of the function, but even then most of the time the results between R and statsmodels seem to be in unison (and it seems that statsmodels is slightly faster than Rpy)

I have two classes that implement unconditional and conditional multivariate density and bandwidth estimation called unconditional_bw and conditional_bw. For unconditional estimation you specify the training data, variable types (c: cont, u:unordered, o:ordered) and bandwidth selection method (you can also specify kernels but I am using defaults: Gaussian, WangVanRyzin and AitchisonAitken)

unconditional_bw(tdat=[c1,u],var_type='cu',bwmethod='normal_reference')

the method unconditional_bw.pdf(edat) will estimate the density at the estimation data

Similarly for conditional estimation you specify dependent and indepenent variables, variable type and bwmethod:

conditional_bw(tydat=[c1,u],txdat=[c2], dep_type='cu',indep_type='c',bwmethod='normal_reference')

Any comments and suggestions on the structure are welcome and appreciated!

George

[josef...@gmail.com]

That's quite a big commit, it will take a bit of time to work our way
through it.
Also we need to see how we can get a reference branch for your work.

Looks like a good way into your GSoC.

Thanks,

Josef

>
> George