build vs. host requirements

[Thomas Holder]

I'm trying to fully understand the new distinction between "build" and "host" requirements which was added in conda 3. After reading the documentation [1], my conclusion was that all libraries (for linking) belong into the "host" section, and all programs which are executed during building belong into "build". My questions and/or observations (with conda-build 3.2.0 on Linux):

1) python can be "build" (executed in build.sh) and "host" requirement (linking CPython extensions). Should it be listed twice? Current AnacondaRecipes don't do that.

2) $PYTHON is only defined if listed in "host". As an example, the recipe for pyqt [2] fails for that reason, it lists python in "build" and uses $PYTHON in build.sh

3) I couldn't find documentation for the "cdt()" function. The CDT packages are headers and libraries, so I would put them into "host". However, the compiler only finds those headers if listing the packages in "build".

Thanks,
Thomas

[1] https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#requirements-section
[2] https://github.com/AnacondaRecipes/pyqt-feedstock

[Ray Donnelly]

Hi Thomas,

1) If you need Python to take part in the build process (e.g. Qt needs python 2.7 for QtDeclarative) then it goes in build, if you do not need it for this then it does not.

Using $PYTHON is cleaner for the case when you have Python in both build and host. We recently changed the activation order so that build comes before host in PATH (I believe this is more correct).

2) PyQt is out of date against the new changes. I am refreshing our Qt stack to 5.9.4 at present and will get round to it soon. Please do not draw too many conclusions about how things should be done from the current state of that recipe.

3) CDT packages stand for "Core Dependency Tree" packages. They are always noarch packages and always belong in build because they are used by our cross compilers and those are build tools and are therefore in the build section. The files in the CDT packages are unpacked into the cross compiler sysroot directory so that the compilers find them without needing to be explicitly told where to look for them. Essentially they are proxies for system libraries. To be exact, they are repackaged RPMs from CentOS6. Doing it this way gives us (and anyone else using our compilers and CDT packages) a good degree of isolation from your actual Linux distribution without requiring us (or you) to use chroots or docker. It also opens up the possibility to do actual cross-compilation though we've not exploited this capability too much.

Packages in the host section usually end up introducing runtime dependencies due to run_exports. If a host package does not use run_exports then you will need to add the package to requirements/run explicitly. Packages in the build section do not introduce any runtime dependencies. It is expected that you will use your system package manager to provide those. They are generally only needed for packages that use OpenGL or X11 which we have decided not to package. We would like in the future for conda to be able to advise on which system package to install when we have used a CDT package during build and also possibly how to install it, e.g. (please ask a system administrator to install some system packages via "dnf install libX11")

Does this answer your questions?

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/conda/DCF7D8A2-1927-4855-BAA4-F378D7FA8259%40schrodinger.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[Ray Donnelly]

> Packages in the build section do not introduce any runtime dependencies

I should clarify, there *is* a way to make this happen but it only needs to be done in very exceptional circumstances. We call these 'strong run_exports". They are used, for example, so that libstdc++ gets added to the runtime requirements when you use {{ compiler('cxx') }} in as a build requirement.

[Thomas Holder]

Hi Ray,

Thanks a lot, this was very helpful! I was wondering whether "host" vs. "build'" makes a difference for "run_exports", and I had already drawn the wrong conclusion because I missed the "strong run_exports" part.

It would be fantastic if your detailed answer finds its way into the documentation! :-)

Cheers,
Thomas

> On Mar 7, 2018, at 12:09 PM, Ray Donnelly <rdon...@anaconda.com> wrote:
>
> > Packages in the build section do not introduce any runtime dependencies
>
> I should clarify, there *is* a way to make this happen but it only needs to be done in very exceptional circumstances. We call these 'strong run_exports". They are used, for example, so that libstdc++ gets added to the runtime requirements when you use {{ compiler('cxx') }} in as a build requirement.
>
> On Wed, Mar 7, 2018 at 11:06 AM, Ray Donnelly <rdon...@anaconda.com> wrote:
> Hi Thomas,
>
> 1) If you need Python to take part in the build process (e.g. Qt needs python 2.7 for QtDeclarative) then it goes in build, if you do not need it for this then it does not.
> Using $PYTHON is cleaner for the case when you have Python in both build and host. We recently changed the activation order so that build comes before host in PATH (I believe this is more correct).
> 2) PyQt is out of date against the new changes. I am refreshing our Qt stack to 5.9.4 at present and will get round to it soon. Please do not draw too many conclusions about how things should be done from the current state of that recipe.
> 3) CDT packages stand for "Core Dependency Tree" packages. They are always noarch packages and always belong in build because they are used by our cross compilers and those are build tools and are therefore in the build section. The files in the CDT packages are unpacked into the cross compiler sysroot directory so that the compilers find them without needing to be explicitly told where to look for them. Essentially they are proxies for system libraries. To be exact, they are repackaged RPMs from CentOS6. Doing it this way gives us (and anyone else using our compilers and CDT packages) a good degree of isolation from your actual Linux distribution without requiring us (or you) to use chroots or docker. It also opens up the possibility to do actual cross-compilation though we've not exploited this capability too much.
> Packages in the host section usually end up introducing runtime dependencies due to run_exports. If a host package does not use run_exports then you will need to add the package to requirements/run explicitly. Packages in the build section do not introduce any runtime dependencies. It is expected that you will use your system package manager to provide those. They are generally only needed for packages that use OpenGL or X11 which we have decided not to package. We would like in the future for conda to be able to advise on which system package to install when we have used a CDT package during build and also possibly how to install it, e.g. (please ask a system administrator to install some system packages via "dnf install libX11")
>
> Does this answer your questions?
>
> On Wed, Mar 7, 2018 at 10:51 AM, Thomas Holder <thomas...@schrodinger.com> wrote:
> I'm trying to fully understand the new distinction between "build" and "host" requirements which was added in conda 3. After reading the documentation [1], my conclusion was that all libraries (for linking) belong into the "host" section, and all programs which are executed during building belong into "build". My questions and/or observations (with conda-build 3.2.0 on Linux):
>
> 1) python can be "build" (executed in build.sh) and "host" requirement (linking CPython extensions). Should it be listed twice? Current AnacondaRecipes don't do that.
>
> 2) $PYTHON is only defined if listed in "host". As an example, the recipe for pyqt [2] fails for that reason, it lists python in "build" and uses $PYTHON in build.sh
>
> 3) I couldn't find documentation for the "cdt()" function. The CDT packages are headers and libraries, so I would put them into "host". However, the compiler only finds those headers if listing the packages in "build".
>
> Thanks,
> Thomas
>
> [1] https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#requirements-section
> [2] https://github.com/AnacondaRecipes/pyqt-feedstock

--

Thomas Holder
PyMOL Principal Developer
Schrödinger, Inc.

[Michael Sarahan]

Thanks Ray for the detailed answer.  Like our software, our docs, are open-source.  This particular doc page is part of https://github.com/conda/conda-docs

and within that repo, the file is at https://github.com/conda/conda-docs/blob/master/docs/source/user-guide/tasks/build-packages/define-metadata.rst

Thomas, it would be very helpful if you take Ray's information and help us clarify our docs.  Since we wrote them and we have all the knowledge behind them, it's not always easy for us to understand how to improve them.

Best,

Michael

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/conda/92BC67BA-9EAF-4056-B038-177C120F0E2E%40schrodinger.com.

[Thomas Holder]

> On Mar 7, 2018, at 7:42 PM, Michael Sarahan <msar...@anaconda.com> wrote:
>
> Thomas, it would be very helpful if you take Ray's information and help us clarify our docs. Since we wrote them and we have all the knowledge behind them, it's not always easy for us to understand how to improve them.

OK. I'll work on a pull request.

[Michael Sarahan]

Thanks!  Looking forward to it!  Feel free to ping me - @msarahan

[Thomas Holder]

My attempt to improve the docs:
https://github.com/conda/conda-docs/pull/560

Cheers,
Thomas