Peer-reviewed package repository for scikits on binstar

[Travis Oliphant]

Hey all, 

One of the things that makes R very nice for users is CRAN.   One big feature of CRAN that is missing from the SciPy ecosystem is that people "submit" packages to CRAN for review.  

binstar.org is a new free service that Continuum is rolling out which makes it possible for anyone to host their own pypi or conda package repositories. 

I was thinking that NumFOCUS could create a repository on binstar.org that scientific-related packages could submit to for inclusion.   Basically, this would allow NumFOCUS to play a role of peer-review for technically related software.   The peer-review would focus on at least the existence of tests, documentation, and ability to build the software, and perhaps even a more thorough code-review for what the software does, whether it overlaps with another project, and how well it is implemented (if we have enough volunteers to review the code). 

Thoughts? 

-Travis

--

Travis Oliphant

Continuum Analytics, Inc.

http://www.continuum.io

[Aaron Meurer]

These are their policies. http://cran.us.r-project.org/. It sounds like part of it is an automated check. 

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "NumFOCUS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numfocus+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Benjamin Root]

This would be an amazing feature to have, but it will have to really mobilize the community around it. If packages are left languishing in their reviews, then that would be problematic.

One question that comes to mind is, why not work to improve PyPi to have this feature?

[Travis Oliphant]

There are two reasons to have this done outside PyPI

 * One reason is that numfocus is not technically python-only.   It would be more encouraging to other languages if we were not tied to Python only. 

*  The second reason is to control our own destiny.   The scientific crowds influence and interaction with the build and packaging community of Python has not been encouraging over the past years and even Guido suggested we just do it ourselves.   I, for one, am not enthusiastic about engaging with the PyPI developers to implement such a feature --- especially when something like binstar exists.  

But, to be clear, I'm more interested in the organization of the peer-review process than where the hosting occurs.   

-Travis

On Mon, Sep 16, 2013 at 10:48 AM, Benjamin Root <ben....@ou.edu> wrote:

This would be an amazing feature to have, but it will have to really mobilize the community around it. If packages are left languishing in their reviews, then that would be problematic.

One question that comes to mind is, why not work to improve PyPi to have this feature?

--
You received this message because you are subscribed to the Google Groups "NumFOCUS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numfocus+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Matthew Rocklin]

I like the idea of third party certification and building a trusted brand.  

I think that it would be good to find a way to incentivize review somehow, perhaps just by prominently publishing their level of involvement.

I don't think that this should become a cultural requirement within the development community.  Adoption of prototypical software by adventurous users drives the ecosystem.  Adding barriers to that process could be detrimental.  Clearly though this would be of use to enterprise users.

I think that a large amount of good could be done automatically; this is good because it bypasses cultural barriers (can I find people to review this?)  Relevant attributes might be the following

*  Does it have a license?  

*  If it's pure Python then what is its test coverage level?  

*  Does it pass its own tests on various architectures/python versions?  

*  How many stars does it have on Github?  

These attributes are all automatable and are usually sufficient to secure my confidence in a small library.

On Mon, Sep 16, 2013 at 8:48 AM, Benjamin Root <ben....@ou.edu> wrote:

This would be an amazing feature to have, but it will have to really mobilize the community around it. If packages are left languishing in their reviews, then that would be problematic.

One question that comes to mind is, why not work to improve PyPi to have this feature?

[Matthew Turk]

On Mon, Sep 16, 2013 at 11:58 AM, Travis Oliphant <tra...@continuum.io> wrote:
> There are two reasons to have this done outside PyPI
>
> * One reason is that numfocus is not technically python-only. It would be
> more encouraging to other languages if we were not tied to Python only.
>
> * The second reason is to control our own destiny. The scientific crowds
> influence and interaction with the build and packaging community of Python
> has not been encouraging over the past years and even Guido suggested we
> just do it ourselves. I, for one, am not enthusiastic about engaging with
> the PyPI developers to implement such a feature --- especially when
> something like binstar exists.
>
> But, to be clear, I'm more interested in the organization of the peer-review
> process than where the hosting occurs.

I think this is an interesting prospect, although I wonder how fine a
line there is between peer reviewing software packages and peer
reviewing journal articles. One thing to consider is how previous
efforts in this arena have fared -- not just within R, but also at the
Insight Journal ( http://www.insight-journal.org/ ), Open Research
Computation ( http://www.openresearchcomputation.com/ , which a number
of Pythonistas were involved with before the journal shuttered),
RunMyCode and so on.

If this goes beyond just a check of the contents of a package -- say,
something similar to what "Cheesecake" did (
https://pypi.python.org/pypi/Cheesecake/0.6.1 ) then perhaps it would
be worth exploring. Otherwise, maybe just metrics for source packages
that could be run would be sufficient. I would say it would need to
rise above automated classification, but I'm not certain how feasible
that is based on the other efforts that have been made in this area
and their successes.

[Travis Oliphant]

Thanks for the excellent feedback, Matthew. 

I completely agree that you don't want to *require* such review.   This is mainly a filtering mechanism so that less-adventurous users can have a collection they start with.   In some sense Anaconda and Canopy provide this sort of filtering already, but it would be nice to have a community-supported and backed collection that was not tied to a single company's decisions. 

It is true that we could do something in a mostly automated fashion.     I actually wish the science was more like software --- people publish papers on whatever they want sans peer-review, but there exist "filtered" publications that actually do traditional peer-review on those otherwise available publications.  In fact, several levels of peer-review would be possible.   One is more of a crowd-sourced review ("stars", citations, etc.).  The other is more traditional --- known experts provide their review.    

This would be similar to how the U.S. Congress used to be (with Representatives elected by the people and the Senate elected by the State legislators). 

-Travis

[Benjamin Root]

Crazy idea...

Make available some sort of Github webservice that project maintainers can easily add to their project's github page to treat the peer-review process like a PR?

Cheers!
Ben Root

[Nathan Goldbaum]

Hi all,

One technical concern with respect to binstar is the need for binary packages across multiple platforms.  If there were a NumFOCUS or Continuum Analytics sponsored build service that automated the job of building on different python, numpy, and OS permutations, that would significantly ease the monetary and time burden of release maintenance.

Second, I'd like to add a word of caution about github-centric criteria for inclusion. Why exclude projects that host on a different service or self-host their repository? As a mercurial user I have some personal stake in this. 

-Nathan

--

[Nathaniel Smith]

On Mon, Sep 16, 2013 at 6:04 PM, Nathan Goldbaum <natha...@gmail.com> wrote:
> Hi all,
>
> One technical concern with respect to binstar is the need for binary
> packages across multiple platforms. If there were a NumFOCUS or Continuum
> Analytics sponsored build service that automated the job of building on
> different python, numpy, and OS permutations, that would significantly ease
> the monetary and time burden of release maintenance.

My impression is that yeah, this is probably the #1 benefit provided
by CRAN -- in R, the equivalent of 'pip install' on Windows just
works, even for packages whose authors don't have Windows.

From the outside it's pretty hard to tell how much review actually
goes on at CRAN. AFAICT from following R-devel, the CRAN owners have
dealt with the popularity of their service by retreating into a
fortress of solitude and refusing to talk to anyone. People regularly
complain about not understanding what CRAN actually checks or being
unable to pre-check their packages before submitting them to CRAN, and
these messages get squashed hard on the grounds that any discussion
whatsoever of CRAN is off-topic for R-devel, that the R core
developers have no influence on it, that there does not exist any
public location where it is on-topic. Pretty weird actually...

Not that it really matters what CRAN does, review is still awesome!

I'm not sure what the benefit of linking review to distribution is,
but I can definitely see how a "review match-makers" service would be
a valuable and welcome service for a lot of developers who have
smaller packages and some general idea that they want to distribute
better code, but aren't really clear on how to do it. If someone
wanted to take the lead on this the MVP wouldn't be much more than a
mailing list, some publicity, and a willingness to spend time on the
logistics of tracking volunteers. A more advanced version could
include building up some documents on how to review, checklist for
quality software (automated tests HOWTO, code coverage checking
HOWTO), a little badge participants could put on their Github (or
other) page next to their little green Travis and coveralls buttons,
etc. Make the rule be that the price of having your package reviewed
is that you have to review someone else's package.

-n

[Benjamin Root]

On Mon, Sep 16, 2013 at 1:25 PM, Nathaniel Smith <n...@pobox.com> wrote:

I'm not sure what the benefit of linking review to distribution is,
but I can definitely see how a "review match-makers" service would be
a valuable and welcome service for a lot of developers who have
smaller packages and some general idea that they want to distribute
better code, but aren't really clear on how to do it. If someone
wanted to take the lead on this the MVP wouldn't be much more than a
mailing list, some publicity, and a willingness to spend time on the
logistics of tracking volunteers. A more advanced version could
include building up some documents on how to review, checklist for
quality software (automated tests HOWTO, code coverage checking
HOWTO), a little badge participants could put on their Github (or
other) page next to their little green Travis and coveralls buttons,
etc. Make the rule be that the price of having your package  reviewed
is that you have to review someone else's package.

-n

Maybe not a direct "quid-quo-pro", as that has degenerate cases, but maybe a stack-overflow-esque karma system?

Ben Root

[Benjamin Root]

On Mon, Sep 16, 2013 at 1:04 PM, Nathan Goldbaum <natha...@gmail.com> wrote:

Second, I'd like to add a word of caution about github-centric criteria for inclusion. Why exclude projects that host on a different service or self-host their repository? As a mercurial user I have some personal stake in this. 

-Nathan

Sorry, I didn't mean to make it seem github-centric. Rather, it would be a way to hook into a project's existing workflow. I know it is possible with github. Does bitbucket offer a similar type of webservice API that projects can add?

Ben Root

[Nathaniel Smith]

Well, I'm thinking it'd be on the honor system, but yeah, this is part
of why there needs to be someone committed to keeping things running
and figuring out sensible solutions for when things occasionally go
pear-shaped.

Homo sapiens comes standard with a pretty decent social interactions
system; you don't really need anything else until you hit scaling
limits. At that point stack-overflow-like stuff becomes useful, but
it's a ways off, and doesn't need to be part of the MVP.

-n

[James Bergstra]

In some sense, the proposal at hand is for more review to take place than currently happens, and for that review to be coordinated, reported, etc.  I don't think shiny stars and karma are going to cover up what sounds like what will ultimately be friction on everyone's current workflow. I've never used R much: what's the incentive? What's the efficiency gain?  It sounds a bit like the proposal is to move toward a centralized peer-review system while most scientific communities are moving toward more distributed ones because central peer review is *inefficient*.

- James

[David Cournapeau]

On Tue, Sep 17, 2013 at 1:49 PM, James Bergstra <james.b...@gmail.com> wrote:

In some sense, the proposal at hand is for more review to take place than currently happens, and for that review to be coordinated, reported, etc.  I don't think shiny stars and karma are going to cover up what sounds like what will ultimately be friction on everyone's current workflow. I've never used R much: what's the incentive? What's the efficiency gain?  It sounds a bit like the proposal is to move toward a centralized peer-review system while most scientific communities are moving toward more distributed ones because central peer review is *inefficient*.

It is only inefficient if not automated.

As was already mentioned, the main value of CRAN is automatic build + availability of binary installers for windows. IMO, a good system is closer to what travis-ci does now, except with windows support, and with the ability to build binary installers.

With wheels, I hope that more and more people will push for binaries on pypi, so I see pypi + travis-ci close to a good solution already.

David

[Skipper Seabold]

On Mon, Sep 16, 2013 at 1:25 PM, Nathaniel Smith <n...@pobox.com> wrote:

On Mon, Sep 16, 2013 at 6:04 PM, Nathan Goldbaum <natha...@gmail.com> wrote:
> Hi all,
>
> One technical concern with respect to binstar is the need for binary
> packages across multiple platforms.  If there were a NumFOCUS or Continuum
> Analytics sponsored build service that automated the job of building on
> different python, numpy, and OS permutations, that would significantly ease
> the monetary and time burden of release maintenance.

My impression is that yeah, this is probably the #1 benefit provided
by CRAN -- in R, the equivalent of 'pip install' on Windows just
works, even for packages whose authors don't have Windows.

From the outside it's pretty hard to tell how much review actually
goes on at CRAN. AFAICT from following R-devel, the CRAN owners have
dealt with the popularity of their service by retreating into a
fortress of solitude and refusing to talk to anyone. People regularly
complain about not understanding what CRAN actually checks or being
unable to pre-check their packages before submitting them to CRAN, and
these messages get squashed hard on the grounds that any discussion
whatsoever of CRAN is off-topic for R-devel, that the R core
developers have no influence on it, that there does not exist any
public location where it is on-topic. Pretty weird actually...

Not that it really matters what CRAN does, review is still awesome!

I wondered this as well. I just reached out to a fairly prominent R package author and asked. My question was a little leading -- how rigorous is the peer review? Does it guarantee correctness/quality or is it "This isn't spam. It builds. The tests pass."

Unless it's going into JSS or the R Journal, then it's almost certainly the latter.

 

I'm not sure what the benefit of linking review to distribution is,
but I can definitely see how a "review match-makers" service would be
a valuable and welcome service for a lot of developers who have
smaller packages and some general idea that they want to distribute
better code, but aren't really clear on how to do it. If someone
wanted to take the lead on this the MVP wouldn't be much more than a
mailing list, some publicity, and a willingness to spend time on the
logistics of tracking volunteers. A more advanced version could
include building up some documents on how to review, checklist for
quality software (automated tests HOWTO, code coverage checking
HOWTO), a little badge participants could put on their Github (or
other) page next to their little green Travis and coveralls buttons,
etc. Make the rule be that the price of having your package  reviewed
is that you have to review someone else's package.

Just an idea, though one with significant overhead. Two levels of package review. The quality review would be linked with a(n open, electronic?) journal article. Incentives all around?

Skipper

[Olivier Grisel]

2013/9/17 David Cournapeau <cour...@gmail.com>:

>
>
>
> On Tue, Sep 17, 2013 at 1:49 PM, James Bergstra <james.b...@gmail.com>
> wrote:
>>
>> In some sense, the proposal at hand is for more review to take place than
>> currently happens, and for that review to be coordinated, reported, etc. I
>> don't think shiny stars and karma are going to cover up what sounds like
>> what will ultimately be friction on everyone's current workflow. I've never
>> used R much: what's the incentive? What's the efficiency gain? It sounds a
>> bit like the proposal is to move toward a centralized peer-review system
>> while most scientific communities are moving toward more distributed ones
>> because central peer review is *inefficient*.
>
>
> It is only inefficient if not automated.
>
> As was already mentioned, the main value of CRAN is automatic build +
> availability of binary installers for windows. IMO, a good system is closer
> to what travis-ci does now, except with windows support, and with the
> ability to build binary installers.
>
> With wheels, I hope that more and more people will push for binaries on
> pypi, so I see pypi + travis-ci close to a good solution already.

+1, although we need to convince the pypi maintainers that being able to
upload OSX binary wheel package as well as windows binaries.

--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel

[Nathaniel Smith]

On Tue, Sep 17, 2013 at 1:49 PM, James Bergstra
<james.b...@gmail.com> wrote:

> In some sense, the proposal at hand is for more review to take place than
> currently happens, and for that review to be coordinated, reported, etc. I
> don't think shiny stars and karma are going to cover up what sounds like
> what will ultimately be friction on everyone's current workflow. I've never
> used R much: what's the incentive? What's the efficiency gain? It sounds a
> bit like the proposal is to move toward a centralized peer-review system
> while most scientific communities are moving toward more distributed ones
> because central peer review is *inefficient*.

Yeah, I can't see a mandatory code review program going anywhere, but
I don't think anyone's proposing that. But if our goal is to generally
increase the quality of code and the quality of programmers in our
ecosystem, then encouraging more code review to happen is a really
good way to do that. Code review always makes code better, and the
people doing it and the people receiving it both always learn
something, and that's true even if you don't have super-expert
programmers involved. And this is especially true for an ecosystem
like ours, which has lots of neophyte programmers working alone, many
of whom are very aware of wanting to improve quality but not being
sure how, and not really having contacts with more experienced
experts/mentors to learn from. So I think there could be a lot of
value in advertising a central place to match up people like this and
get review happening - the bang-for-buck of coordinating something
like this seems pretty high to me.

-n

[Benjamin Root]

Perhaps it would be a good idea to put together a "best practices" or "intro" guide to help newbies get ramped up. A best practice guide on directory structure, packaging, unit tests, docs, command-line parsing? Note, I am talking about real basic stuff that should give most people a good leg up on getting their work ready for this repository. Right now, I have never really felt that such documentation exists anywhere except in a very scattered, non-cohesive way.

Am I somewhat volunteering myself for such a task? Possibly. For the Fall season, I am heading up a workplace effort in getting our research staff ramped up on these "essentials". However, it does start to bring us back over to a python-centric focus that I know Travis wants to avoid.

Thoughts?

Ben Root

[Nathaniel Smith]

On Tue, Sep 17, 2013 at 5:12 PM, Benjamin Root <ben....@ou.edu> wrote:
> Perhaps it would be a good idea to put together a "best practices" or
> "intro" guide to help newbies get ramped up. A best practice guide on
> directory structure, packaging, unit tests, docs, command-line parsing?
> Note, I am talking about real basic stuff that should give most people a
> good leg up on getting their work ready for this repository. Right now, I
> have never really felt that such documentation exists anywhere except in a
> very scattered, non-cohesive way.

I think this would be very useful.

> Am I somewhat volunteering myself for such a task? Possibly. For the Fall
> season, I am heading up a workplace effort in getting our research staff
> ramped up on these "essentials". However, it does start to bring us back
> over to a python-centric focus that I know Travis wants to avoid.

I think if that's an issue then the solution is to search for
volunteers to write similar documents for other languages, not for
anyone to avoid writing one for Python :-).

-n

[Jacob Barhak]

Hi Travis, and all NumFocus 

First - count me as a volunteer to promote this. This is important - regardless of details. 

Travis has a good point. Review should be open. The review can be accomplished simultaneously by:

1. Volunteers

2. Specific invitation from organizers

3. Dedicated critic - probably with pay

The type of review should be noted, yet all types above should be allowed simultaneously - as long as there is a name, affiliation, funding, and conflict of interest deceleration attached to the review. Blind review should not be an option since it can have hidden interests foreign to the open source community. If someone disagrees please explain. 

No model is perfect, yet the open model made possible due to recent technological advances have not been fully explored by existing publishing systems. It is important to explore these paths. It is important to move away from unreproducible science based on scientific honor system and peer review of papers to reproducible science based on code and tests. Describing the idea in lay terms or complex text is still important, yet providing tested procedures others can easily follow is imperative to build higher on the shoulders of others.

I strongly agree with James Bergstra. Current centralized publishing systems are insufficient - it is time for a change. 

Travis has the correct vision here.

       Jacob

Sent from my iPhone

[Radim Řehůřek]

No model is perfect, yet the open model made possible due to recent technological advances have not been fully explored by existing publishing systems. It is important to explore these paths. It is important to move away from unreproducible science based on scientific honor system and peer review of papers to reproducible science based on code and tests. Describing the idea in lay terms or complex text is still important, yet providing tested procedures others can easily follow is imperative to build higher on the shoulders of others.

I strongly agree with James Bergstra. Current centralized publishing systems are insufficient - it is time for a change. 

Travis has the correct vision here.

This is a beautiful idea: a decentralized publishing platform, based around reproducibility.

It's something I've been thinking about as well, but found too daunting. I could offer some volunteering time with the development/integration though, if it ever comes to that (backend/frontend non-designer stuff: web dev, python, javascript...) Or reviews in my niche domain (data mining&information retrieval).

That is, as long as this is for the general good of scientific computing and/or Python, and not just for some pet projects of NumFOCUS. I'm still not clear who NumFOCUS, or this particular platform proposal, is really for, based on my exchanges on this mailing list.

Radim

[denis...@t-online.de]

Travis, folks,

  Have any reviews come along since September ?

(There are of course quite different kinds of reviews, by / for different people,
newbies / grad students / experts --
-- doc reviews: top of my list
    (stumbled across https://github.com/PharkMillups/beautiful-docs
    but that's way too long, a grab bag)
-- code reviews: dunno.
    http://codereview.stackexchange.com/questions/tagged/python
    has quite a few improve-my-first-python-program.
-- area overviews, e.g. 2d graphics packages or Cython generators
-- mechanizable: list Klines of code, Klines test, pages of doc, requires, has-a-wiki
    trivial, useful, hasn't happened.)

Summary: a couple of concrete reviews to discuss would be a start.


(On concrete: Knuth gave a class in "Concrete mathematics" at Stanford years ago --
some civil engineers snuck out of the first class.)