topic weeks

[josef...@gmail.com]

statsmodels has too many moving parts, not enough moving parts, too many issues, too many gaps

and not enough topic experts.

It would be possible to make significant progress with a concentrated effort during some dedicated time such as one to a few weeks.

The idea is to allocate topic weeks where all priority and effort goes into making progress on issues related to one topic and merge as fast as possible.

This would be more fun and more productive if more developers and contributors would collaborate. Similar to a sprint but more spread out in terms of location and time. For me this would also have the advantage of reducing start-up time for getting into a topic, which is much easier if it is concentrated in time than when I have to switch to a topic to reply to and look into an issue.

There are a huge number of topics that would be available, either in fixing, improving and enhancing current models or in adding new or almost finished new functionality.

One of the current problems for statsmodels developers is the lack of reviewers and of maintainers. This has as consequence, for example, that we loose contributions because I don't have the time to see a contribution through until it's merged and until it is in a state that would require relatively low future maintenance. For contributors it would be helpful if it's possible to plan in advance and being able to have high expectation that there will be a mergable result.

Concentrating the effort would improve this in many cases.

Caveats: 

This will not help enough for very large projects or topics. For example GEE took more than half a year of work. All GSOC projects are more than 3 or 4 months of work with several weeks just for review and getting ready for merge. Some projects even take more than one year of GSOC.

Other cases with possible problems are topics where the design is not clear, and it takes time to figure out what needs to be or should be done. Prime example for this right now are adding weights. The basic properties are relatively easy after several years of discussion, but the details especially for all the extra results that we have for our models is difficult.

However, I think topic weeks would be a great help for cases where we have or can find "digestible" tasks.

Josef

[Ralf Gommers]

This seems like a good idea. Browsing the open PRs, there are a large amount that can be merged with fairly small effort (especially if the reviewer has been looking at the code/submodule already recently) it looks like.

I don't have much bandwidth for the next month, but if there's a build/maint topic-weeks at some point I'm happy to help.

Ralf

[Chad Fulton]

I also think it's a good idea. I like the idea of a short, defined time frame, like a week. It would also be great if they were planned / announced a fair ways in advance. That might make it more likely that I would be able to participate in more projects that I don't necessarily already have expertise in.

[Brock Mendel]

I'll echo Chad on this one.  The point about start-up time is compelling.

[josef...@gmail.com]

On Fri, Sep 22, 2017 at 10:06 PM, Chad Fulton <chadf...@gmail.com> wrote:

On Fri, Sep 22, 2017 at 7:06 PM, Ralf Gommers <ralf.g...@gmail.com> wrote:

On Sat, Sep 23, 2017 at 3:51 AM, <josef...@gmail.com> wrote:

statsmodels has too many moving parts, not enough moving parts, too many issues, too many gaps

and not enough topic experts.

It would be possible to make significant progress with a concentrated effort during some dedicated time such as one to a few weeks.

The idea is to allocate topic weeks where all priority and effort goes into making progress on issues related to one topic and merge as fast as possible.

This would be more fun and more productive if more developers and contributors would collaborate. Similar to a sprint but more spread out in terms of location and time. For me this would also have the advantage of reducing start-up time for getting into a topic, which is much easier if it is concentrated in time than when I have to switch to a topic to reply to and look into an issue.

There are a huge number of topics that would be available, either in fixing, improving and enhancing current models or in adding new or almost finished new functionality.

One of the current problems for statsmodels developers is the lack of reviewers and of maintainers. This has as consequence, for example, that we loose contributions because I don't have the time to see a contribution through until it's merged and until it is in a state that would require relatively low future maintenance. For contributors it would be helpful if it's possible to plan in advance and being able to have high expectation that there will be a mergable result.

Concentrating the effort would improve this in many cases.

Caveats: 

This will not help enough for very large projects or topics. For example GEE took more than half a year of work. All GSOC projects are more than 3 or 4 months of work with several weeks just for review and getting ready for merge. Some projects even take more than one year of GSOC.

Other cases with possible problems are topics where the design is not clear, and it takes time to figure out what needs to be or should be done. Prime example for this right now are adding weights. The basic properties are relatively easy after several years of discussion, but the details especially for all the extra results that we have for our models is difficult.

However, I think topic weeks would be a great help for cases where we have or can find "digestible" tasks.

This seems like a good idea. Browsing the open PRs, there are a large amount that can be merged with fairly small effort (especially if the reviewer has been looking at the code/submodule already recently) it looks like.

Some of it is also coming up with a solution that works for a set of open issues with related problems.

eg. I have been looking into discrete for some time, mainly because of the count model GSOC

https://github.com/statsmodels/statsmodels/issues/3925  (next on my todo list)

https://github.com/statsmodels/statsmodels/issues/3924 (open)

https://github.com/statsmodels/statsmodels/pull/3921 (just merged)

Several bug reports on github or stackoverflow helped to identify the problems and discuss or bounce around ideas. The actual implementation takes just a few days, even with version problems across unit test environments.

 

I don't have much bandwidth for the next month, but if there's a build/maint topic-weeks at some point I'm happy to help.

There are at least these two that sound important, but I haven't kept up with recent development in these areas.

https://github.com/statsmodels/statsmodels/pull/3195

https://github.com/statsmodels/statsmodels/pull/3791

Your experience with scipy might help in finishing and merging them or some version of them that fulfill the same usecase.

 

Ralf

I also think it's a good idea. I like the idea of a short, defined time frame, like a week. It would also be great if they were planned / announced a fair ways in advance. That might make it more likely that I would be able to participate in more projects that I don't necessarily already have expertise in.

There are a large number of topics and many of them will need some planning in advance. One question is how many would possibly participate or sign up, which would affect how much preparation we need.

For example in time series forecasting we have pull requests for holt-winters, ewm style models and tools. They need review, finishing up and merging, or decisions on which version to prepare for merging. I think this would take less than a week of actual review and coding if several contributors participate.

Relate idea is to have a issue/bug fix week on a topic. This combines subtopics that wouldn't necessarily overlap in the code, but would overlap in becoming familiar with a topic or model. E.g. one week of reviewing all open ARIMA question and make decisions or fix as many as possible. The problem is again setup cost, it takes time to figure out or remember how AR/ARMA/ARIMA works even before working on any changes. 

I will need one to two weeks with Evgeny and whoever else is interested to finish up the last part of his GSOC with several additional count models.

Similar, survey design GSOC needs at least one or two weeks of concentrated work to merge.

Some older unmerged topics like GAM and penalized spline GSOC will take me just a week to get back into the topic, and I'm not sure we have any other contributors interested in this. Similar for the Mixed GLM GSOC.

James Morton and I were privately discussing topic weeks and a sprint for multivariate methods, especially with respect to reuse for compositional data. That's a big topic and needs planning to put it into digestible parts that will result in piecewise addition of new functionality. At the current stage I have only vague ideas about any implementation design or issues.

Related to this, MNLogit has several post estimation results that don't work because params is 2-D. jbrockmendel was also looking into it and it will be a high priority topic week to fix it and prepare the setup for more multivariate (2-D endog) models.

There are a lot more topics which in the past were either one or two contributor/maintainer topics. I don't know if there would be enough interest to make an organized topic week. Some of this is working on my own backlog.

My hope is that if several developers push a topic at the same time, it is much more likely to make progress and get something to merge, than when different developers work on the same topic but months or years apart.

Another personal advantage for developers, contributors or users is that it is a good time to become familiar with a model or group of models when several developers work or look at the same model at the same time.

Josef

[Brock Mendel]

> There are a large number of topics and many of them will need some planning in advance. One question is how many would possibly participate or sign up, which would affect how much preparation we need.

Coming at the question from the other direction: how many people (person-hours?) need to sign up in order for the idea to reach escape velocity?  Would the participants of this thread be enough to do a trial run?


How do you envision splitting the goals/labor between new features vs improving/fixing/cleaning/testing/performance/issue-closing/...?  I'd be perfectly happy with focusing on the latter and removing dead code, while others may not find that all that motivating.

[josef...@gmail.com]

On Sat, Sep 23, 2017 at 9:00 PM, Brock Mendel <jbrock...@gmail.com> wrote:

> There are a large number of topics and many of them will need some planning in advance. One question is how many would possibly participate or sign up, which would affect how much preparation we need.

Coming at the question from the other direction: how many people (person-hours?) need to sign up in order for the idea to reach escape velocity?  Would the participants of this thread be enough to do a trial run?

Minimal size is one person, two would be the usual for most of our PRs, but concentrated in time, 3 to 4 would be good, more than 4 needs good preparation.

Person-hours depend on the size of the topic and is often difficult to guess. MNLogit should be a few days for two developers/contributors. Some of the stalled PRs might require many more person-hours and might not be feasible in a week or two unless there are enough participants, or several topic weeks are used.

Improved time series forecasting support contains many, largely separate pieces and could be anything from 2 to 10 person weeks, I guess.

I want to emphasize that contributing is not only writing code. If someone is interested in review, making or running examples and test cases or provide background information, then this would also be a great contribution. For some topics figuring out what should or needs to be done is more work than the actual coding.

 

How do you envision splitting the goals/labor between new features vs improving/fixing/cleaning/testing/performance/issue-closing/...?  I'd be perfectly happy with focusing on the latter and removing dead code, while others may not find that all that motivating.

Fixing insufficient unit tests are always a good target. For the rest it is mostly a trade-off between how large the benefits are and how large the costs are. There are some highly desired and demanded new features or high priority fixes that I never got into and never got resolved because my personal setup cost are too high, mainly when it's not my area of expertise or because it would take me a long time to work my way through the issues.

Other issues are new infrastructure features that we need to have and I spend a lot of time trying to figure out what to do. Example again is weights, we need them for survey, variance modelling or heteroscedasticity, inverse probability weighting and robust estimation. All these are slowed down or need workarounds because of the missing weights support across models.

Another infrastructure issue that is still more open is multivariate support in the generic methods, eg. for MNLogit, VAR/VECM, and MultivariateOLS.

Better support for time series forecasting is one of the most demanded features. But it's not really an area where I have much practical experience, and I'm only interested in getting more than superficially involved in it, if other people carry the main weight and make the main decisions.

One of the long term benefits, if this is successful, would be that I don't need to jump into and across so many topics. It would be great if there are more sub fields within statsmodels that have local experts where I don't have to be the default maintainer/reviewer anymore and consequently not anymore the bottleneck for those areas.

aside on issue-closing

IMO closing issues is similar to line coverage in unit tests.

Both are imperfect measures about the underlying health of the code. For example, regression tests or, even worse, smoke tests prevent some refactoring mistakes and show that the code runs, but they don't "verify" the code and the results. Even full line coverage doesn't mean that we have tested all options and inherited methods.

Similarly, closing issues just so that they are closed after fixing the specific issue or bug does not tell anything about the underlying structural problems or about possible better solutions. I often leave issues open or open two new issues for every one issue that I close, as a reminder that we need a better solution or better structure or better generic handling. Sometimes after seeing several related issues, a generic or structural solution or new common pattern becomes easier to figure out.

(Point against this: I recently started to do my github search more often in all issues including closed issues instead of just searching open issues so I can also see what previous related fixes have been made.)

Josef

[Kevin Sheppard]

Too many basic PRs languish for way too long.  This puts off contributors making useful fixes, such as doc errors. 

Doc errors should be merged as soon as seen unless there is something fundamentally wrong. 

Small, almost trivial PRs should be merged quickly (little not review).  

Small, tested PRs should also be merged quickly.

[Ralf Gommers]

On Mon, Sep 25, 2017 at 12:29 AM, Kevin Sheppard <kevin.k....@gmail.com> wrote:

Too many basic PRs languish for way too long.  This puts off contributors making useful fixes, such as doc errors. 

very true

Doc errors should be merged as soon as seen unless there is something fundamentally wrong. 

Small, almost trivial PRs should be merged quickly (little not review).  

Small, tested PRs should also be merged quickly.

One other suggestion: when sending PRs, consider maintainer load. If you already have 10 open, maybe review someone else's PR instead of sending number 11.

Ralf