Extending conda-build

[Floris Bruynooghe]

Hi,

I was wondering what the attitude would be to making conda-build extensible.  I imagine being able to add more configuration to the meta.yaml file and being able to be called at certain points.  Maybe it is clearer if I explain my uses cases for which I'd like to build tools at some point:

1) A conda-watch tool with a source/watch stanza which is a regex of a URL.  This would work very similar to the debian/watch facility in that I'd be able to invoke "conda watch recipe" and it would check if a new release is available upstream.

2) A tool to check all shared libraries which are used to detect if any shared libraries are not present in the conda environment but accidentally link with a system library.  This could be done explicitly by invoking a tool in the test/commands section but would be much nicer if it could be automatically called when the build is finished.

3) A new about/license_files setting which lists files to be automatically installed into share/doc/$package/.

All these are probably not general-purpose enough to warrant getting them into conda-build itself, they can all be implemented completely outside of conda-build and meta.yaml but for consistency would be a lot nicer to work with if they could hook into it.

So what is your opinion on providing hooks or similar in conda-build and making meta.yaml extensible?

Regards,

Floris

[Aaron Meurer]

On Tue, Feb 17, 2015 at 5:22 AM, Floris Bruynooghe <fl...@devork.be> wrote:
> Hi,
>
> I was wondering what the attitude would be to making conda-build extensible.
> I imagine being able to add more configuration to the meta.yaml file and
> being able to be called at certain points. Maybe it is clearer if I explain
> my uses cases for which I'd like to build tools at some point:
>
> 1) A conda-watch tool with a source/watch stanza which is a regex of a URL.
> This would work very similar to the debian/watch facility in that I'd be
> able to invoke "conda watch recipe" and it would check if a new release is
> available upstream.
>
> 2) A tool to check all shared libraries which are used to detect if any
> shared libraries are not present in the conda environment but accidentally
> link with a system library. This could be done explicitly by invoking a
> tool in the test/commands section but would be much nicer if it could be
> automatically called when the build is finished.

This one actually is something that it would be nice to have in conda
build itself. We already have conda inspect linkages on Linux, which
makes it easy to do this manually.

>
> 3) A new about/license_files setting which lists files to be automatically
> installed into share/doc/$package/.
>
> All these are probably not general-purpose enough to warrant getting them
> into conda-build itself, they can all be implemented completely outside of
> conda-build and meta.yaml but for consistency would be a lot nicer to work
> with if they could hook into it.
>
> So what is your opinion on providing hooks or similar in conda-build and
> making meta.yaml extensible?

How would the extensibility work?

Aaron Meurer

>
> Regards,
> Floris
>
> --
> You received this message because you are subscribed to the Google Groups
> "conda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to conda+un...@continuum.io.
> To post to this group, send email to co...@continuum.io.
> Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Thomas Caswell]

+1 from me on #2

[Travis Oliphant]

For #2, there is some experimental code that Trent Nelson was working on that is still in a branch that did this for Linux. 

-Travis

Travis Oliphant

CEO

Continuum Analytics, Inc.

http://www.continuum.io

[Floris Bruynooghe]

On 17 February 2015 at 16:57, Aaron Meurer <aaron....@continuum.io> wrote:

> So what is your opinion on providing hooks or similar in conda-build and
> making meta.yaml extensible?

How would the extensibility work?

So given that I'm an py.test developer the following is heavily inspired by how it's plugin system works.  It's may also remind of emacs' hooks if you're familiar with them.

I imagine when you install the plugin/helper it somehow gets registered with conda, py.test uses setuptools entrypoints for this.  Then conda could define a number of hooks which get called at certain points.  For the examples earlier in the thread the things needed to do are:

* At startup call a hook which can register new settings in meta.yaml, this could throw an exception if the setting is already taken meaning that extensions can't take names from conda-build (at the expense of breaking hard on upgrades).

* At other times the hook API can even be simpler, conda-build just needs to call a registered function at the right times maybe with some helpful arguments, e.g.:

- after meta.yaml parsing

- before creating the workspace

- after populating the workspace

- after creating the build environment

- before starting the build script

- after the build script finished

- before the test is executed

- right at the end

This list is quite loose for now and some of them might be the same point.  I'm just trying to give a feel of what could work.

As an aside, py.test is actually looking at splitting off it's plugin system into a separate project.  So if you like the hook system it may or may not be interesting to use this (it would get me some motivation to get that split finished ;-))  http://pytest.org/latest/plugins.html gives a bit more of a feel how hooks work in py.test.

Let me know what you think of this approach and if you'd like me to describe this way it could work a bit more concrete.

Regards,

Floris

[Floris Bruynooghe]

On 17 February 2015 at 16:57, Aaron Meurer <aaron....@continuum.io> wrote:

On Tue, Feb 17, 2015 at 5:22 AM, Floris Bruynooghe <fl...@devork.be> wrote:
> 2) A tool to check all shared libraries which are used to detect if any
> shared libraries are not present in the conda environment but accidentally
> link with a system library.  This could be done explicitly by invoking a
> tool in the test/commands section but would be much nicer if it could be
> automatically called when the build is finished.

This one actually is something that it would be nice to have in conda
build itself. We already have conda inspect linkages on Linux, which
makes it easy to do this manually.

Oh, that is nice.  I didn't know this existed!

[jelle feringa]

+1

Great idea