Let's learn Julia

[josef...@gmail.com]

https://github.com/JuliaStats/GLM.jl/blob/master/LICENSE.md
and especially this
https://github.com/dmbates/MixedModels.jl/blob/master/LICENSE.md

It might be easier for R guys to figure out the dispatch than for us.

Josef
<I like classes>

[josef...@gmail.com]

On Sat, May 11, 2013 at 3:46 PM, <josef...@gmail.com> wrote:
> https://github.com/JuliaStats/GLM.jl/blob/master/LICENSE.md
> and especially this
> https://github.com/dmbates/MixedModels.jl/blob/master/LICENSE.md
>
> It might be easier for R guys to figure out the dispatch than for us.

or how about a MIT licensed .rda dataset reader

https://github.com/HarlanH/DataFrames.jl/blob/15bac952f6d20c74311ef20fa7487329b3498e16/src/RDA.jl

Looks like some R developers are moving into Julia

Josef


>
> Josef
> <I like classes>

[josef...@gmail.com]

or this one: a port of Vincent's Rdatasets into Julia

https://github.com/johnmyleswhite/RDatasets.jl (GPL-3)

I like the last part :

'''
Licensing and Intellectual Property

Following Vincent's lead, we have assumed that all of the data sets in
this repository can be made available under the GPL-3 license. If you
know that one of the datasets released here should not be released
publicly or if you know that a data set can only be released under a
different license, please contact me so that I can remove the data set
from this repository.
'''

Josef

[Padarn Wilson]

It does indeed seem like Julia has a backing in the R/Stats community. I think some of the major core developers from R have moved over to working on Julia, which is probably encouraging this.

What do you make of it? Do you know much Julia?

[Padarn Wilson]

Sorry about the formatting of this reply, I'm having a little trouble getting it to behave properly.
 

[josef...@gmail.com]

I don't know much about Julia, I just looked at it a bit last year
when it showed up on some mailing lists.
I am, at the current stage, also not keeping track of numba, and the
other attempts to get faster code execution in python.

I didn't know that some R developers moved to Julia. The main point of
what I just saw is that they write statistics under MIT license in
code that is almost readable. Being able to look over the sholders of
Bates to implement mixed models could be quite useful.
Besides the matlab fileexchange and Boost (which is impossible to read
for me), it would be the third larger area where we could benefit
(copy) from license compatible code.


my minimal impression about Julia:

When reading the code, Julia looks a bit like cython with type definitions.
In the mixed models code, it looks nice that they can directly call
into Blas/Lapack, while that's still difficult with cython.

It sounds fast, but (except for maybe the nicer Blas/Lapack access), I
don't think there is much reason to switch to it compared to using
more cython.
There might be a future where we delegate some numerics to Julia
instead of cython and c (or fortran).


However, I lost all interest in Julia, after I saw that they don't
have namespaces and classes.
Greg Ewing makes the point
http://mail.python.org/pipermail//cython-devel/2012-April/002376.html
and next messages in the thread.
http://mail.scipy.org/pipermail/nipy-devel/2012-April/007480.html
They added namespace modules in the mean time (the ticket was closed).

Instead of classes everything works by dispatch, which, I think, is
not a good way to organize any larger body of code.
I cannot image how statsmodels would look like without classes (maybe
I can when I think about some matlab code I wrote.:)

So my soothsayer prediction: Julia is doomed as general purpose
programming language since neither namespaces nor classes are in the
basic design.

(But Julia is still young and can still grow some features.)


about statsmodels:
I think it's still more important that we expand enough that we can do
all the basic statistics and econometrics in python, than getting
every ounce of speed.
cython, and maybe some other speed improvements, are important, and
will become more important for us, when our code takes seconds or
minutes, while the other packages give you instantaneous answers.

(
I still prefer algorithmic improvements when they are available to
coding boring loops in cython/C, but if numpy and scipy don't have the
fast loops, we will have to do it ourselves.

https://github.com/scipy/scipy/pull/526
https://github.com/statsmodels/statsmodels/issues/449
Skipper just mentioned in a pull request that adding analytical
hessians and gradients gave a big speed improvement compared to
numerical derivatives.
)

Josef
being too wordy, as usual

[Padarn Wilson]

Wow, thanks for the detailed response. Interesting to hear your thoughts.

I think the direction you're taking with statsmodels is smart: Building a broad base of functionality which have efficient underlying algorithms. Projects like Numba, Theano etc are exciting IMO, but I'm also hesitant to spend much time on them because it really isn't clear what will be the best approach to getting speed in Python a few years down the track.

I'm very interested in the statsmodels package, and have been meaning to find time to look into adding some bootstrap/non-parametric functionality. However, I think my main concern is whether or not it is going to be able to gain traction among the people who are actually doing a lot of statistics. This is why the interest in Julia from the Statistics/R community is interesting.

But I see now that you're original link was more about the licence, rather than Julia itself. Sorry for half hijacking your discussion!

Cheers,

Padarn

[josef...@gmail.com]

roughly my view on gaining traction for statsmodels:

statsmodels and python still has problems attracting (academic)
statisticians, even though the original stats.models was written by a
statisticians. Julia might be more familiar to R developers than
python (pure guessing). And academic statisticians and their students
have been focused for a long time on R.

However, our strategy to world domination (i.e. getting a large enough
market share to have a critical, sustainable mass) goes through
application areas where python is already partially established or
where python is starting to get a foothold.

I think the success of statsmodels and statistics in python will not
come because statisticians or econometricians want to write their new
algorithms in it. (We are not at that stage yet, but I hope we get
there eventually, especially in econometrics.)
The success of statsmodels comes because we are part of an
infrastructure of packages.

Especially, the success of pandas for datahandling means that there
are widely used data structures that statsmodels supports.

There are many areas where python has a strong presence (brain
research, psychology, spatial data analysis - pysal, finance,
astronomy, ...). Python is large enough in some of those fields that
they have their own datastructures and statistics, but I have seen
several of them switching to pandas, and maybe they will also use more
statsmodels in the future.

Then, there are many fields and users that need some statistics and
econometrics, but where statistics is not important enough to dominate
the choice of programming languages.

Thomas' work is very important to introduce python for statistics to a
general audience
http://work.thaslwanter.at/Stats/html/

Kevin Sheppard wrote a toolbox in matlab (financial econometrics), but
now has an introduction to python
http://www.kevinsheppard.com/wiki/Python_for_Econometrics

statsmodels is still missing a few crucial pieces, especially mixed
models and repeated measures.
But, another year or two, and we will have most of the crucial parts.
(We have two GSOC candidates that would be expanding statsmodels into
new areas.)

scikit-learn and the other machine learning packages:
They have a different focus and don't have to compete as much as we do
with packages that are 20 or 30 years old. However, their success at
the same time as we develop statsmodels, means that users that need
both machine learning and statistics, will be able to switch packages
but not the programming language.


asides:
Another, once more fashionable programming language, with an important
flaw in design or convention (attitude) among programmers
http://stackoverflow.com/questions/717506/if-monkey-patching-is-permitted-in-both-ruby-and-python-why-is-it-more-controve
Changing the core behavior of a language might be useful for fast
(web) development where you have mainly your own programs to worry
about, but, it makes an awful system for building something like
SciPy, a larger infrastructure of related library programs, when
everyone can pull the rug out from under your feet.
There are also some Statistics in Ruby packages that I check every
once in a while.


rpy could be a serious competition to or complement for statsmodels,
however, it never build a larger developer base, and it became too
low-level, making it too difficult to maintain it as cross-platform
package. (speaking as Windows user/developer who never made the
transition from rpy to rpy2.)


The main reason why python will win in the long run and is gaining a
lot of traction in science is that it is more "fun" to code in python
than in any of the other languages (especially for developers that are
not professional programmers), and because python can stay close
enough in capabilities and performance with specialized languages that
that is not a reason to avoid or drop python.

Josef

[josef...@gmail.com]

On Sun, May 12, 2013 at 4:36 PM, Laurent Gautier <lgau...@gmail.com> wrote:
>
>
> On Sunday, May 12, 2013 4:42:00 PM UTC+2, josefpktd wrote:
>>
>> (...)

>>
>>
>>
>> rpy could be a serious competition to or complement for statsmodels,
>> however, it never build a larger developer base, and it became too
>> low-level, making it too difficult to maintain it as cross-platform
>> package. (speaking as Windows user/developer who never made the
>> transition from rpy to rpy2.)
>
>
>

> The development of Rpy has stopped several years ago (probably before
> statmodels even started), and I am not sure to follow the argument about
> "becoming too low-level, making it too difficult too maintain as
> cross-platform".

The first test suite of scikits.statsmodels was almost completely
written with rpy, which was at the time just a little bit outdated but
I could run it on Windows.

>
> Regarding rpy2, I think that the relatively small number of developers is
> because of the energy barrier (requires good knowledge of R, R's C API,
> Python's C API, Python - not so many people at the intersection of the
> four), but this is improving (ohloh does not seem to flag us as "too small
> number of developers") and the ratios "number of downloads"/"number of
> developers" and "number of open bugs"/"number of developers" suggest that
> this is not concern.
>
> Support for Windows has become non-existent because of the absence of
> developers meeting the requirements above for that platform (so
> self-inflicted by the Windows community ;-) ).

I didn't say rpy2 is not successful, and ipython has an interface to
R, which as far as I know is not available on Windows.

The C interface (as I understand it) between R, R's C API, Python's C
API, and Python is what I think of as too low level. Coding a raw C
interface between two different programming languages (without the
help of something like cython) sounds too difficult to me as a cross
platform strategy (unless you have the resources of a commercial
distribution).
(Sounds a bit like "self-inflicted" absence of Windows developers
because the barriers are too high.)

Seems to me that that was your choice. I'm not saying that it was a
bad choice (given your objectives), but it makes it useless to me, and
statsmodels cannot use it to close some gaps, nor is it an alternative
to statsmodels among Windows users.

For statsmodels, I'm fighting all the time to keep a very good
availability on Windows, and I'm definitely not in the camp of the
"why don't you get a virtual Linux machine?" developers. Which is the
response potential users get when they try to install python libraries
that have some dependencies that work almost only on Linux.
Nevertheless, the vast majority of statsmodels developers work on Linux.

Josef

>
> L.

[Padarn Wilson]

I've been partially involved in the development of a tsunami simulation package, which is cross platform and uses a mix of Python and raw C. You're entirely correct - maintaining usability on Windows is a huge time sink, but completely necessary for us.

[Padarn Wilson]

Just a thought on this. I recently was able to ask the manager of the pycogent package for bioinformatics: http://pycogent.org/, about moving much of his statistics and data handling to things like pandas and statsmodels. His main reason not to do so was the fear of adding dependencies - especially with an anticipated move to Python 3.x.

I think something that would really help the uptake of these general packages in Python would be 1) Complete compatibility with Python 3 and 2) Some sort of lightweight (if possible) multi-platform distribution.

[josef...@gmail.com]

>> There are many areas where python has a strong presence (brain
>> research, psychology, spatial data analysis - pysal, finance,
>> astronomy, ...). Python is large enough in some of those fields that
>> they have their own datastructures and statistics, but I have seen
>> several of them switching to pandas, and maybe they will also use more
>> statsmodels in the future.
>
>
> Just a thought on this. I recently was able to ask the manager of the
> pycogent package for bioinformatics: http://pycogent.org/, about moving much
> of his statistics and data handling to things like pandas and statsmodels.
> His main reason not to do so was the fear of adding dependencies -
> especially with an anticipated move to Python 3.x.
>
> I think something that would really help the uptake of these general
> packages in Python would be 1) Complete compatibility with Python 3 and 2)
> Some sort of lightweight (if possible) multi-platform distribution.

1) we have had python 3 compatibility for 2 years now, pandas about
the same, patsy since before we started to require it, IIRC.
(statsmodels will follow scipy and numpy to switch to a single
codebase for python 2 and 3 later this year.)

2) there are source or binary packages for all major distributions,
pythonxy for Ubuntu has even a daily builds.
pandas and matplotlib are besides numpy and scipy in the SciPyStack
which for Windows also includes statsmodels and patsy (Gohlke), and
all pandas, statsmodels and patsy are just a pip or easy_install.
Skipper wrote the scripts that we even have daily (as needed) Windows
binaries that are automatically build.

The only restriction that I can see is that all recent pandas require
numpy >= 1.6 which shouldn't be too tough to satisfy.
Other than, that we are pretty good in supporting several version of
the different dependencies.

So far I haven't seen much of a demand for a special mini distribution
for the data analysis in python packages, but the idea shows up every
once in a while.

We (packages related to this) are all getting pretty good support from
distribution specific developers, Yaroslav for Debian, Tim for Ubuntu
and Christoph for Windows.
I'm not familiar with the distributions for Mac, but, as I understand,
Chris Fonnesbeck has the install scripts for this area, and
statsmodels is in macports.

This is really useful for us and very much appreciated, pythonxy is
doing daily (when there are changes) testing of master on Ubuntu,
Debian has a large variety of machines that is useful to catch
cross-machine problems (big-endian, small endian) and Christoph is
running the tests of several packages when new numpy and scipy
releases come out, and catches if some compatibility problems have
sneaked in (as long as they are covered by our test suite).

maybe another answer that's too long, but I think it's pretty cool how
the system works.

Josef

[Denis A. Engemann]

Back to the initial subject: Josef, thanks for sharing this, that's really exciting! It's also easier to read bates' code in Julia than in R + C --- bookmarked.

Denis

[Laurent Gautier]

On Sunday, May 12, 2013 11:39:11 PM UTC+2, josefpktd wrote:

On Sun, May 12, 2013 at 4:36 PM, Laurent Gautier <lgau...@gmail.com> wrote:
>
>
> On Sunday, May 12, 2013 4:42:00 PM UTC+2, josefpktd wrote:
>>
>> (...)
>>
>>
>>
>> rpy could be a serious competition to or complement for statsmodels,
>> however, it never build a larger developer base, and it became too
>> low-level, making it too difficult to maintain it as cross-platform
>> package. (speaking as Windows user/developer who never made the
>> transition from rpy to rpy2.)
>
>
>
> The development of Rpy has stopped several years ago (probably before
> statmodels even started), and I am not sure to follow the argument about
> "becoming too low-level, making it too difficult too maintain as
> cross-platform".

The first test suite of scikits.statsmodels was almost completely
written with rpy, which was at the time just a little bit outdated but
I could run it on Windows.

Rpy2 had Windows binaries throughout the 2.0.x series. 

 

>
> Regarding rpy2, I think that the relatively small number of developers is
> because of the energy barrier (requires good knowledge of R, R's C API,
> Python's C API, Python - not so many people at the intersection of the
> four), but this is improving (ohloh does not seem to flag us as "too small
> number of developers") and the ratios "number of downloads"/"number of
> developers" and "number of open bugs"/"number of developers" suggest that
> this is not concern.
>
> Support for Windows has become non-existent because of the absence of
> developers meeting the requirements above for that platform (so
> self-inflicted by the Windows community ;-) ).

I didn't say rpy2 is not successful, and ipython has an interface to
R, which as far as I know is not available on Windows.

The C interface (as I understand it) between R, R's C API,  Python's C
API, and Python is what I think of as too low level. Coding a raw C
interface between two different programming languages (without the
help of something like cython) sounds too difficult to me as a cross
platform strategy (unless you have the resources of a commercial
distribution).

Bridging at the C level 2 languages implemented in C and offering C-level interfaces is the natural to proceed, I think. 

Alternatives to that are covered in two other Python packages:

- pyper (child process communicating with pipes)

- pyrserve (communicate with Rserve - you'll require an Rserve server)

 

(Sounds a bit like "self-inflicted" absence of Windows developers
because the barriers are too high.)

rpy2 is (reportedly) compiling on Windows but contributing back binary builds does not seem up on the list for people who do (very few people kindly did in the past, but eventually moved on).  

 

Seems to me that that was your choice. I'm not saying that it was a
bad choice (given your objectives), but it makes it useless to me, and
statsmodels cannot use it to close some gaps, nor is it an alternative
to statsmodels among Windows users.

I am sorry to be unable to provide help. I do include patches for Windows whenever contributed, but this is up to the community of users to provide for the less compiling inclined (debian binary packages are often within hours after I release without me doing anything beside announcing the release).

 

For statsmodels, I'm fighting all the time to keep a very good
availability on Windows, and I'm definitely not in the camp of the
"why don't you get a virtual Linux machine?" developers. Which is the
response potential users get when they try to install python libraries
that have some dependencies that work almost only on Linux.
Nevertheless, the vast majority of statsmodels developers work on Linux.

This is going beyond the scope rpy2 (and drifting OT), but the root of the problem is the burden of setting up build environments requiring the compilation of C code on Windows. For example, you'll remember the hunt for no-longer-available version of MSVC++ required to compile Python packages with C extensions.

[Jan Schulz]

Hi!

On 13 May 2013 23:12, Laurent Gautier <lgau...@gmail.com> wrote:
> rpy2 is (reportedly) compiling on Windows but contributing back binary
> builds does not seem up on the list for people who do (very few people
> kindly did in the past, but eventually moved on).

There are builds for windows available at the "usual place" :-)

http://www.lfd.uci.edu/~gohlke/pythonlibs/#rpy2

Kind regards,

Jan

[Nathaniel Smith]

On 13 May 2013 17:12, "Laurent Gautier" <lgau...@gmail.com> wrote:
>
>
>
> On Sunday, May 12, 2013 11:39:11 PM UTC+2, josefpktd wrote:
>> For statsmodels, I'm fighting all the time to keep a very good
>> availability on Windows, and I'm definitely not in the camp of the
>> "why don't you get a virtual Linux machine?" developers. Which is the
>> response potential users get when they try to install python libraries
>> that have some dependencies that work almost only on Linux.
>> Nevertheless, the vast majority of statsmodels developers work on Linux.
>
>
> This is going beyond the scope rpy2 (and drifting OT), but the root of the problem is the burden of setting up build environments requiring the compilation of C code on Windows. For example, you'll remember the hunt for no-longer-available version of MSVC++ required to compile Python packages with C extensions.

Wine works well for this. If you get the environment set up once you can save it as a tgz or whatever in case it gets messed up, or write a script to set it up reproducibly. This is what numpy/scipy do for release builds (you could even steal their release scripts), and I used to do it for xpra... It's not really very hard and totally automateable.

-n

[Laurent Gautier]

Wow ! Awesome. I didn't know about it.

 

[josef...@gmail.com]

I didn't know about it either. Thanks Jan for the pointer.
Christoph to the rescue !

I tried out the win32 version with my older R 2.12 wih py27 and a few
examples that I ran worked without problems.
(after setting R_HOME and R_USER)

The only smaller problem I have seen so far, is that I'm getting
"Unable to unlink tempfile" warnings, and some files are left behind
in the Windows temp folder.

Josef

>