developing / debugging conda recipes

[Chris Barker]

Hi all,

I'm workign no a conda recipe at the moment, and finding it really annoyin got have to re-run the whole thing from the start each time ai change something. I end up with a workflow like:

write a minimal recipe.

try to build it

it errors out.

fix the error

try to build it

repeat (sometimes a lot!)

then:

add a feature

try to build it

it errors out.

and so on for a while.

Maybe I'm just incompetent, but I have to repeat this edit/build cycle a LOT, so the fact that the build cycle take a while is painful.

Is there a way to "stash" the build partway through, so I can run it faster?

And/or a way to have conda build get started, set itself up, then leave me in a shell with everyting set, so I can edit adn re-run the build.sh script without re-doing the whole setup?

any hints appreciated.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Jon Wilson]

A failed build usually leaves a special env behind named "work" (I think).  That env in the context in which build.sh runs.  So you can activate that env and run whatever you like.  Look around in miniconda/conda-bld and miniconda/envs/work after a failed build.

One of the actual conda experts will probably correct me, especially since I'm working from vague memory of having done this several months ago.
Regards,
Jon

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Chris Barker]

On Fri, Jul 31, 2015 at 2:35 PM, Jon Wilson <jsw...@gmail.com> wrote:

A failed build usually leaves a special env behind named "work" (I think).  That env in the context in which build.sh runs.  So you can activate that env and run whatever you like.  Look around in miniconda/conda-bld and miniconda/envs/work after a failed build.

Perfect, thanks --  trying that

I found a _build environment, maybe that's it.

however, once activating it, I find that PREFIX isn't defined -- so I can't run build.sh and have it work the same way.

and, now that I think about it -- I need the dir where the source tarball has been unpacked, etc. The workign dir of the build process:

found it -- there is a "work" dir in:

miniconda/conda-bld/work 

so I can get myself there, but still don't have the env variables set -- almost there, though. I also noticed the cache dir, so at least it's not re-dowloading each time...

-CHB

-Chris

[Phil Elson]

The work directory is not the _build directory:

 * work is the directory where the source is put, and where configure etc. is executed

 * _build is the prefix where a successful build would put the files to install

work can be found in <conda build root>/work  (from memory) and it also has the full build script which you can execute (complete with environment variables).

HTH

[Ilan Schnell]

Yes, that is correct the "work" directory, usually located at <root prefix>/conda-bld/work, contains the source tree of the package to be build, and "build.sh" is executed with this directory as the current working directory.

_build is the build conda environment, where the build dependencies get installed into and the build.sh installs files into.  Within "build.sh", PREFIX is set to this location, e.g. /Users/ilan/anaconda/envs/_build.  This is what allows you to say "./configure --prefix=$PREFIX" within your build.sh.

_test is the conda environment for testing the package.  That is, once the conda package is created (from the files added into _build by build.sh), it is tested here.  This is deliberately a different directory (from _build), in order to catch hard-coded paths in the binaries in the test.

[Chris Barker]

On Sat, Aug 1, 2015 at 4:17 PM, Ilan Schnell <il...@continuum.io> wrote:

Yes, that is correct the "work" directory, usually located at <root prefix>/conda-bld/work, contains the source tree of the package to be build, and "build.sh" is executed with this directory as the current working directory.

_build is the build conda environment, where the build dependencies get installed into and the build.sh installs files into.  Within "build.sh", PREFIX is set to this location, e.g. /Users/ilan/anaconda/envs/_build.  This is what allows you to say "./configure --prefix=$PREFIX" within your build.sh.

_test is the conda environment for testing the package.  That is, once the conda package is created (from the files added into _build by build.sh), it is tested here.  This is deliberately a different directory (from _build), in order to catch hard-coded paths in the binaries in the test.

thanks guys - this is satrting to make sense. Except:

""" it also has the full build script which you can execute (complete with environment variables)."""

where do I find this? Or do I have to set the environment variable myself. 

Now that I think about, myabe "PREFIX" is all I need, so not hard to do.

-CHB

[Jon Wilson]

On 08/03/2015 10:45 AM, Chris Barker wrote:

On Sat, Aug 1, 2015 at 4:17 PM, Ilan Schnell <il...@continuum.io> wrote:

Yes, that is correct the "work" directory, usually located at <root prefix>/conda-bld/work, contains the source tree of the package to be build, and "build.sh" is executed with this directory as the current working directory.

_build is the build conda environment, where the build dependencies get installed into and the build.sh installs files into.  Within "build.sh", PREFIX is set to this location, e.g. /Users/ilan/anaconda/envs/_build.  This is what allows you to say "./configure --prefix=$PREFIX" within your build.sh.

_test is the conda environment for testing the package.  That is, once the conda package is created (from the files added into _build by build.sh), it is tested here.  This is deliberately a different directory (from _build), in order to catch hard-coded paths in the binaries in the test.

thanks guys - this is satrting to make sense. Except:

""" it also has the full build script which you can execute (complete with environment variables)."""

where do I find this? Or do I have to set the environment variable myself. 

Now that I think about, myabe "PREFIX" is all I need, so not hard to do.

I might suggest adding a "printenv" early in your build script.  This way you can see the exact environment.
Regards,
Jon

[Chris Barker]

On Mon, Aug 3, 2015 at 8:48 AM, Jon Wilson <jsw...@gmail.com> wrote:

I might suggest adding a "printenv" early in your build script.  This way you can see the exact environment.

great idea, thanks!

-CHB 

[Aaron Meurer]

When this happens I usually just cd to the source in the work
directory, activate the _build directory from the last build (or just
use absolute paths), and try to work through the build in the shell,
writing everything down in the build.sh as I do it. When I think I
have something that works, or when I need to "clean" the build
environment or work directory, I'll re-build the recipe.

Beyond that, my only suggestion (which I've never done, but which
ought to work) would be to save partially built states somehow, either
through backing up the work directory or through some kind of version
control.

Aaron Meurer

[Aaron Meurer]

On Mon, Aug 3, 2015 at 10:45 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Sat, Aug 1, 2015 at 4:17 PM, Ilan Schnell <il...@continuum.io> wrote:
>>
>> Yes, that is correct the "work" directory, usually located at <root
>> prefix>/conda-bld/work, contains the source tree of the package to be build,
>> and "build.sh" is executed with this directory as the current working
>> directory.
>>
>> _build is the build conda environment, where the build dependencies get
>> installed into and the build.sh installs files into. Within "build.sh",
>> PREFIX is set to this location, e.g. /Users/ilan/anaconda/envs/_build. This
>> is what allows you to say "./configure --prefix=$PREFIX" within your
>> build.sh.
>>
>> _test is the conda environment for testing the package. That is, once the
>> conda package is created (from the files added into _build by build.sh), it
>> is tested here. This is deliberately a different directory (from _build),
>> in order to catch hard-coded paths in the binaries in the test.
>
>
> thanks guys - this is satrting to make sense. Except:
>
> """ it also has the full build script which you can execute (complete with
> environment variables)."""
>
> where do I find this? Or do I have to set the environment variable myself.

The build script is only copied over to the work directory on Windows.
On Unix the build script is executed in place in the recipe.

Aaron Meurer

[Chris Barker]

On Mon, Aug 3, 2015 at 10:24 AM, Aaron Meurer <aaron....@continuum.io> wrote:

When this happens I usually just cd to the source in the work
directory, activate the _build directory from the last build

will that set PREFIX?

 

(or just
use absolute paths), and try to work through the build in the shell,
writing everything down in the build.sh as I do it. When I think I
have something that works, or when I need to "clean" the build
environment or work directory, I'll re-build the recipe.

perfect, this sounds exactly what I had in mind. As it happens, I've got the build I was working on going, but next time, I'll be sure to give this workflow a try.

-Chris

[Aaron Meurer]

On Mon, Aug 3, 2015 at 1:44 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Mon, Aug 3, 2015 at 10:24 AM, Aaron Meurer <aaron....@continuum.io>
> wrote:
>>
>> When this happens I usually just cd to the source in the work
>> directory, activate the _build directory from the last build
>
>
> will that set PREFIX?

No, you'll also need to set your environment manually. I always just
hard-code the build prefix manually at the command line as I am
figuring things out (like ./configure
--prefix=~/anaconda/envs/_build). At the end of the day, you'll need
to run the full recipe to make sure everything works in a clean
environment. This workflow is mostly about figuring stuff out while
minimizing waiting.

Aaron Meurer

[Chris Barker]

On Mon, Aug 3, 2015 at 11:58 AM, Aaron Meurer <aaron....@continuum.io> wrote:

 

> will that set PREFIX?

No, you'll also need to set your environment manually. I always just
hard-code the build prefix manually at the command line as I am
figuring things out (like ./configure
--prefix=~/anaconda/envs/_build).

I can do an:

export PREFIX=~/anaconda/envs/_build

the run build.sh, yes?

 

At the end of the day, you'll need
to run the full recipe to make sure everything works in a clean
environment. This workflow is mostly about figuring stuff out while
minimizing waiting.

yup -- exactly what I was looking for.

-Chris

 

[Ilan Schnell]

No, don't put

export PREFIX=~/anaconda/envs/_build

into your build.sh.  PREFIX is already set by conda-build when it runs the build.sh.

--

[Chris Barker - NOAA Federal]

No, don't put

export PREFIX=~/anaconda/envs/_build

into your build.sh.  PREFIX is already set by conda-build when it runs the build.sh.

Got it. I didn't mean in the build.sh, I meant outside the build.sh, so I could run and test build.sh outside of conda build, without altering the build script.

-Chris

[Ilan Schnell]

Ah, ok.  I was worried there.

[David Froger]

Hello,

When debugging conda recipe involing long C/C++ compilation, ccache [0] can be
very usefull to speed-up the conda build time.

[0] https://ccache.samba.org/

Quoting Chris Barker (2015-07-31 23:31:56)