Customising a recipe at build time

[Floris Bruynooghe]

Hi,

It would be nice to be able to customise a recipe at build time.  One use case is e.g. to enable or disable applying of branding patches (via jinjan2 templating of the meta.yaml file).  Another would be for enabling bootrapping of an environment in build.sh, e.g. normally you depend on gcc but when bootstrapping you do not.  What I was trying to do was this:

# build.sh

if [ -z "$BUILD_BOOSTRAP" ]; then

    conda create -y --use-local -p reqs gcc

    export PATH=$(pwd)/reqs/bin:$PATH

fi

./configure --prefix=$PREFIX
make
make install

However the environment which builld.sh sees is very tightly controlled and environment variables are not getting passed through.  Is there any mechanism to achieve this which I am missing?

Regards,

Floris

[Floris Bruynooghe]

Hi again,

So I'm quite happy to produce a patch which exposes environment variables to the build script, but it would be nice to know if I'm not missing another way to achieve this.

Regards,

Floris

[Aaron Meurer]

On Thu, Jan 29, 2015 at 9:14 AM, Floris Bruynooghe <fl...@devork.be> wrote:
> Hi,
>
> It would be nice to be able to customise a recipe at build time. One use
> case is e.g. to enable or disable applying of branding patches (via jinjan2
> templating of the meta.yaml file). Another would be for enabling
> bootrapping of an environment in build.sh, e.g. normally you depend on gcc
> but when bootstrapping you do not. What I was trying to do was this:
>
> # build.sh
> if [ -z "$BUILD_BOOSTRAP" ]; then
> conda create -y --use-local -p reqs gcc
> export PATH=$(pwd)/reqs/bin:$PATH
> fi
> ./configure --prefix=$PREFIX
> make
> make install

What exactly is your build process here? Are you planning on
rebuilding gcc with the same version of itself?

I personally would just manage this by bumping build numbers, and keep
the build.sh static.

Aaron Meurer

>
> However the environment which builld.sh sees is very tightly controlled and
> environment variables are not getting passed through. Is there any
> mechanism to achieve this which I am missing?
>
> Regards,
> Floris
>

> --
> You received this message because you are subscribed to the Google Groups
> "conda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to conda+un...@continuum.io.
> To post to this group, send email to co...@continuum.io.
> Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Ilan Schnell]

Hi Floris,

the problem with exposing environment variables to the build script is that the build becomes less reproducible.  If people relay of environment settings on their system (which are not checked into the recipe), someone else might have a hard time building the recipe.  This is the reason they where hidden in the first place.

I see however your need as well.  Maybe a solution could be to use conda 'features'.

- Ilan

[Phil Elson]

On 3 February 2015 at 16:57, Ilan Schnell <il...@continuum.io> wrote:

the problem with exposing environment variables to the build script is that the build becomes less reproducible.

FWIW I believe the Windows conda build environment exposes all environment variables currently - this should probably change in the future.

[Floris Bruynooghe]

On 3 February 2015 at 16:56, Aaron Meurer <aaron....@continuum.io> wrote:

On Thu, Jan 29, 2015 at 9:14 AM, Floris Bruynooghe <fl...@devork.be> wrote:
> It would be nice to be able to customise a recipe at build time.  One use
> case is e.g. to enable or disable applying of branding patches (via jinjan2
> templating of the meta.yaml file).  Another would be for enabling
> bootrapping of an environment in build.sh, e.g. normally you depend on gcc
> but when bootstrapping you do not.  What I was trying to do was this:
>
> # build.sh
> if [ -z "$BUILD_BOOSTRAP" ]; then
>     conda create -y --use-local -p reqs gcc
>     export PATH=$(pwd)/reqs/bin:$PATH
> fi
> ./configure --prefix=$PREFIX
> make
> make install

What exactly is your build process here? Are you planning on
rebuilding gcc with the same version of itself?

So normally you would rebuild the next version of gcc with an earlier version of itself.  But when bootstrapping the environment you don't yet have a gcc package to install, relying on the host to provide gcc instead.  During bootstrapping of your environment there are several other packages which need special care like this.

The above isn't a huge issue, one can simply modify the recipes by hand during the bootstrapping as there's only a good handful ones involved.  However I was also trying to figure out a way how different groups could share the same recipes repository while being interested in slightly different details.  E.g. if we would share the conda-recipes repository then we'd like to be able to apply our own (branding) patches to a recipe.  I was imagining something like:

package:

  name: python

build:

  patches:

    - getpath_symlink_support.diff

    {% if environ['ABILISOFT''] %}

    - abilisoft_branding.diff

    - disable_user_site_packages.diff

    {% endif %]

Maybe this is entirely the wrong approach to this problem and maybe there's something much smarter to be done with VCSes/git to have modifications to a base repository or yet something else.  But that's a usecase I had in mind as well.

Regards,

Floris

[Floris Bruynooghe]

On 3 February 2015 at 16:57, Ilan Schnell <il...@continuum.io> wrote:

the problem with exposing environment variables to the build script is that the build becomes less reproducible.  If people relay of environment settings on their system (which are not checked into the recipe), someone else might have a hard time building the recipe.  This is the reason they where hidden in the first place.

I see however your need as well.  Maybe a solution could be to use conda 'features'.

Yes, I understand your reluctance to expose all environment variables.  It probably is a good idea to not do this.

For conda features I thought one needed two recipes, one for each feature.  Is there a way to build multiple features from a single recipe somehow?  Anyway, that would still seem slightly hackish as you then have to remember to remove the feature after bootstrapping.

Regards,
Floris

[Floris Bruynooghe]

So if I understand it correctly environment variables are exposed to the jinja2 templating.  So maybe it would be nice if the metadata could specify a way of either exposing some environment variables to the build.(sh|bat) script explicitly, e.g.:

build:

  script_env:

    - MY_VAR

    - ANOTHER_VAR

Similar but explicitly controlling the values:

build:

  script_env:

    MY_VAR: {{ environ['MY_VAR'] }}

    ANOTHER_VAR: hello

Another approach to pass information into the build script is to pass extra arguments to the build script invoked, then accessible via $1, $2, etc.  It could look very similar:

build:

  script_args:

    - {{ environ['FOO'] }}

The benefit of this one is that you do not have to worry about overriding environment variables controlled by conda-build.

What are the thoughts on constructs like these?

Regards,
Floris

[Bradley Kreider]

On Tuesday, February 3, 2015 at 11:57:05 AM UTC-5, Ilan wrote:

the problem with exposing environment variables to the build script is that the build becomes less reproducible.  If people relay of environment settings on their system (which are not checked into the recipe), someone else might have a hard time building the recipe.  This is the reason they where hidden in the first place.

If the only exposed ENV variables came from the recipe, wouldn't that be a good compromise?   That is allow a dictionary of env variables to be set in the meta.yaml and have only those items show up in the build.sh.   Jinja templating can make the meta.yaml dynamic, but at least we capture the meta.yaml settings that are used in the final package, right?

-- Bradley

[David Froger]

Hello,

Just to mention another case where customising a recipe at build time may be
usefull.

The use case would be to read the $TRAVIS environment variable to detect
whether or not the package is being build on a travis [1] host.

The problem is that on travis hosts, 2 cores are available [2], but CPU_COUNT
value is 32 [3]. If build.sh contains 'make -j$CPU_COUNT', we make use of
32 cores instead of 2, which yields the C++ process to be killed
(c++: internal compiler error: Killed (program cc1plus).

A simple workaround would be:

if [ "$TRAVIS" == "true" ]
then
CPU_COUNT=2
fi

I'm not sure this is a sane motivation to have access to custom environment
variable, as the real solution would be that Travis exposes CPU_COUNT to 2,
but this allow to quickely implement simple workaournd.

Best regards,
David

[1] https://travis-ci.org
[2] http://docs.travis-ci.com/user/ci-environment/#The-Build-Machine
[3] https://github.com/travis-ci/travis-ci/issues/2407
[4] http://docs.travis-ci.com/user/ci-environment/#Environment-variables


Quoting Floris Bruynooghe (2015-02-04 13:49:33)

[Ilan Schnell]

Hello,

I'm going to go ahead and implement what Floris suggested.  That is specifying a list of environment variables which are copied into the environment of the build script:

build:
  script_env:
    - MY_VAR
    - ANOTHER_VAR

- Ilan

[Floris Bruynooghe]

On 17 February 2015 at 19:03, Ilan Schnell <il...@continuum.io> wrote:

Hello,

I'm going to go ahead and implement what Floris suggested.  That is specifying a list of environment variables which are copied into the environment of the build script:

build:
  script_env:
    - MY_VAR
    - ANOTHER_VAR

Thanks, this will be useful!

Regards,

Floris