Automatically build all pypi packages

[Stefan Otte]

Hey conda community,

TLDR: are there any plans to automatically build conda packages for all/top x pypi packages? This would really improve conda!

Using `conda skeleton` to build dependencies from pypi is pretty straight forward in most cases and does not take up too much time. But I guess everybody is rebuilding dependencies. Sure, some packages are already on binstar. But do you know/trust the user? Is he updating it regularly? (I'm not updating my stuff.)

I think it would be cool if https://binstar.org/pypi would be an authoritative source for all pypi packages. Why aren't we doing it? Are there any technical issues? What's your take on it?

Best,

 Stefan

[Aaron Meurer]

The biggest technical issue would be that we'd need dedicated build
machines for all three operating systems.

I agree that skeleton pypi works pretty well most of the time, and
that this would probably be not too hard to get somewhere with. It's
really just a question of whether or not Continuum wants to dedicate
resources to building this out.

Aaron Meurer

>
>
> Best,
> Stefan
>
> --
> You received this message because you are subscribed to the Google Groups
> "conda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to conda+un...@continuum.io.
> To post to this group, send email to co...@continuum.io.
> Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Chris Barker]

On Wed, Apr 22, 2015 at 8:43 AM, Aaron Meurer <aaron....@continuum.io> wrote:

I agree that skeleton pypi works pretty well most of the time, and
that this would probably be not too hard to get somewhere with. It's
really just a question of whether or not Continuum wants to dedicate
resources to building this out.

Maybe this could be a community effort.

I've found that conda skeleton works MOST of the time, but not quite all, so there does need to be some hand tweaking, or at least checking. Also, it doesn't work reliably on Windows, though mostly you can build the skeleton on OS-X or Linux, and then build on Windows.

So even with build machines, still not totally automagical.

Maybe a semi-official repo on gitHub for conda build scripts -- and a "trusted group of folks with push permission -- that same group of folks would have push permission to the binstar channel.

We're doing this now for Met-Ocean packages under the "IOOS" Banner --

https://github.com/ioos/conda-recipes

and

https://binstar.org/ioos

Note that gitHub integration with CI systems makes this pretty darn cool (when you can get the service you need...)

 it works pretty well. But someone needs to herd the cats.

-Chris

 

Aaron Meurer

>
>
> Best,
>  Stefan
>
> --
> You received this message because you are subscribed to the Google Groups
> "conda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to conda+un...@continuum.io.
> To post to this group, send email to co...@continuum.io.
> Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert McGibbon]

Chris,

We're doing something similar for computational biophysics packages under the "omnia" banner at https://github.com/omnia-md/conda-recipes. We're building py27/33/34 from those recipes that get pushed to binstar for both OS-X and linux-64 (targeting RHEL6 or greater glibc, so we can't do the builds on travis-ci). I don't think we have the automated builds for windows running successfully yet, but someone is working on it.

Maybe there's some way we can work together, or exchange best practices?

-Robert

[Aaron Meurer]

On Wed, Apr 22, 2015 at 2:11 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Wed, Apr 22, 2015 at 8:43 AM, Aaron Meurer <aaron....@continuum.io>
> wrote:
>>
>> I agree that skeleton pypi works pretty well most of the time, and
>> that this would probably be not too hard to get somewhere with. It's
>> really just a question of whether or not Continuum wants to dedicate
>> resources to building this out.
>
>
> Maybe this could be a community effort.
>
> I've found that conda skeleton works MOST of the time, but not quite all, so
> there does need to be some hand tweaking, or at least checking. Also, it
> doesn't work reliably on Windows, though mostly you can build the skeleton
> on OS-X or Linux, and then build on Windows.

On Windows, the issue is with the patch package. If you use the one
that comes with conda it doesn't work. You'll need to use
http://gnuwin32.sourceforge.net/packages/patch.htm.

The only other main issue I know with skeleton is
https://github.com/conda/conda-build/issues/76 (it doesn't work with
packages that use numpy.distutils). Other packages that fail typically
do so because they do something weird in their setup.py (which is
unfortunately a non-trivial percentage of packages). Also a few don't
list dependencies, and some require C library dependencies, so those
would have to be kept track of separately.

Aaron Meurer

[Chris Barker]

On Wed, Apr 22, 2015 at 12:35 PM, Robert McGibbon <rmcg...@gmail.com> wrote:

We're doing something similar for computational biophysics packages under the "omnia" banner at https://github.com/omnia-md/conda-recipes. We're building py27/33/34 from those recipes that get pushed to binstar for both OS-X and linux-64 (targeting RHEL6 or greater glibc, so we can't do the builds on travis-ci). I don't think we have the automated builds for windows running successfully yet, but someone is working on it.

I've helped out with the IOOS packages, but I'm not leading the effort by any means -- I think some of the more core-people on that project monitor this list, though. But a couple notes:

we're using obvious-ci:

https://github.com/pelson/Obvious-CI

to help automate some of this -- and it also helps with getting the conda packages built on a regular workstation as well -- do check it out. And it's got Appveyor working for Windows -- maybe some hints in there.

 

Maybe there's some way we can work together, or exchange best practices?

Good idea -- I would use gitHub to contact the core people -- maybe an issue on the IOOS repo?

It does seem like these efforts are a bit scattered -- it would be nice to share resources more. MAybe think of it this way:

Core generally useful packages for Scientific computing:

  - Continuum is supporting this already

Domain-specific packages:

  - makes sense to have domain specific repos / binstar channels:

      - IOOS for met-ocean stuff, 

      - omnia for biophsyics

There are also App-specific package collections. For instance, we have an app with a good pile of dependencies:

  

https://github.com/NOAA-ORR-ERD/PyGnome

We would like folks to be able to do a one-stop install of everything, without adding who-knows-how-many binstar channels. So I've started:

https://github.com/NOAA-ORR-ERD/orr-conda-recipes

 and an accompanying binstar channel:

https://binstar.org/noaa-orr-erd

(maybe like your omnia set?)

That way, we can tell folks: Install miniconda, add our binstar channel, then install the entire set of dependencies (likely as a conda environment)

So -- would it be good to have a single home for a wider variety of general-purpose packages -- i.e. a good chunk of PyPi? Maybe.

If it could be trusted and maintained, which would require a fair bit of work, and a lot of cat-herding. But, overall, less work than all of us doing it ourselves!

-Chris

[Chris Barker]

On Wed, Apr 22, 2015 at 12:50 PM, Aaron Meurer <aaron....@continuum.io> wrote:

On Windows, the issue is with the patch package. 

 

The only other main issue I know with skeleton is

https://github.com/conda/conda-build/issues/76 (it doesn't work with
packages that use numpy.distutils).

 

Other packages that fail typically
do so because they do something weird in their setup.py

 

 Also a few don't

list dependencies, and some require C library dependencies, so those
would have to be kept track of separately.

Exactly -- what all this points to is that there are enough packages that don't "just work" to prevent a completely automated system. But having folks add packages one by one really isn't so bad, if you have more than a handful of people doing it.

-CHB

[Thomas Caswell]

Some what related to what Chris is working on, at BNL/NSLS-II we are maintaining recipes for x-ray specific packages at https://github.com/NSLS-II/conda-prescriptions.

Tom

--

[Chris Barker - NOAA Federal]

Hmm,

I'm ambivalent  here: 

We have a handful of communities/projects building and supporting Binstar channels to provide specific communities with a "vetted" source for what they need.

I think this works pretty well, and the combination of conda, Binstar, github, and the CI services provides a nice infrastructure to support all that.

On the other hand, there is quiet a bit of duplicated effort, and folks that aren't tied in to a specific supported project are stuck with choosing which of ten different Binstar channels to use for a given package, and have to pull stuff from probably multiple channels.

(Side note -- maybe a way to search Binstar for multiple packages at once and then you could find the channel that supports the largest subset)

So: maybe it would be good to have a semi-official "community" channel, where a handful of folks have push permissions, and we can all merge our efforts. If we took the superset of just the four projects identified on this thread, I'll bet we'd get a pretty good set of packages.

At the moment, I can't see any downside to my putting the packages my users care about into a larger collection, rather than my own set. I would still only be responsible for my subset, so no more work for me, but I'd get help with all the overlapping packages.

With conda's robust versioning, I think this could work well.

-CHB

[Aaron Meurer]

On Sun, Apr 26, 2015 at 11:49 AM, Chris Barker - NOAA Federal
<chris....@noaa.gov> wrote:
> Hmm,
>
> I'm ambivalent here:
>
> We have a handful of communities/projects building and supporting Binstar
> channels to provide specific communities with a "vetted" source for what
> they need.
>
> I think this works pretty well, and the combination of conda, Binstar,
> github, and the CI services provides a nice infrastructure to support all
> that.
>
> On the other hand, there is quiet a bit of duplicated effort, and folks that
> aren't tied in to a specific supported project are stuck with choosing which
> of ten different Binstar channels to use for a given package, and have to
> pull stuff from probably multiple channels.
>
> (Side note -- maybe a way to search Binstar for multiple packages at once
> and then you could find the channel that supports the largest subset)
>
> So: maybe it would be good to have a semi-official "community" channel,
> where a handful of folks have push permissions, and we can all merge our
> efforts. If we took the superset of just the four projects identified on
> this thread, I'll bet we'd get a pretty good set of packages.

Let's give it a try https://binstar.org/community. Thomas, Chris, what
are your Binstar usernames (or anyone else who wants to take part for
that matter)?

By the way, the copy feature of Binstar (which isn't documented super
well, and is only available via the command line client) is great for
this sort of thing. It lets you copy a package from one channel to
another, without having to reupload it.

Aaron Meurer

[Anthony Scopatz]

On Tue, Apr 28, 2015 at 12:09 PM Aaron Meurer <aaron....@continuum.io> wrote:

On Sun, Apr 26, 2015 at 11:49 AM, Chris Barker - NOAA Federal
<chris....@noaa.gov> wrote:
> Hmm,
>
> I'm ambivalent  here:
>
> We have a handful of communities/projects building and supporting Binstar
> channels to provide specific communities with a "vetted" source for what
> they need.
>
> I think this works pretty well, and the combination of conda, Binstar,
> github, and the CI services provides a nice infrastructure to support all
> that.
>
> On the other hand, there is quiet a bit of duplicated effort, and folks that
> aren't tied in to a specific supported project are stuck with choosing which
> of ten different Binstar channels to use for a given package, and have to
> pull stuff from probably multiple channels.
>
> (Side note -- maybe a way to search Binstar for multiple packages at once
> and then you could find the channel that supports the largest subset)
>
> So: maybe it would be good to have a semi-official "community" channel,
> where a handful of folks have push permissions, and we can all merge our
> efforts. If we took the superset of just the four projects identified on
> this thread, I'll bet we'd get a pretty good set of packages.

Let's give it a try https://binstar.org/community. Thomas, Chris, what
are your Binstar usernames (or anyone else who wants to take part for
that matter)?

Please add me too.  'scopatz' I am pretty sure.

[Dan Allan]

I'd like to help too: 'danielballan'

[Aaron Meurer]

I've added you guys. Go nuts.

Aaron Meurer

[Thomas Caswell]

tacaswell

[Chris Barker - NOAA Federal]

Cool! I've been offline for a bit, but will see what I can do.

-Chris

[Matt Craig]

Could I be added too? I'm starting to work on building several astronomy-related packages, which end up pulling in some non-astronomy packages. It all may as well go somewhere useful!

Binstar username is mwcraig

Matt Craig