using pip in conda build

[David Michael Gang]

I know that i can also use pip in conda envirinments, but i want that everything will be managed with conda. There is a module called grpcio (https://github.com/grpc/grpc/tree/master/src/python/grpcio) which can't be built by conda on windows, but i can run pip and it will make all the necessary work. Is there a way to run pip and take the build artifacts and upload them to a conda custom channel ?

[Michael Sarahan]

Tell me more about what you mean by "can't be built by conda on windows."  It may be a lot harder than you would like, and it may require using the MSYS2 tool stack, but if someone built it for Windows and it's on PyPI, then it is possible to build it for conda.

The general issue with using pip to install things for conda packages is that you want only the files related to your package, not any dependencies.  When you see pip used in conda recipes, you'll also see --no-deps as an argument.

The other concern is binary compatibility.  For a very long time, PyPI would not accept linux wheels, because there was no standard for what their binary compatibility should be.  There is now the manylinux1 standard, but unfortunately it is not enforced adequately by PyPI.  Tensorflow illustrates this point well.  Although that team labels their wheels as "manylinux1," they are built on a much newer platform, and are not actually usable on the platforms specified by the manylinux1 standard.  Windows is (strangely) less troublesome in this regard, and repackaging Windows wheels is much less troublesome (though still generally frowned on).

On Sun, Oct 29, 2017 at 3:37 AM, David Michael Gang <micha...@gmail.com> wrote:

I know that i can also use pip in conda envirinments, but i want that everything will be managed with conda. There is a module called grpcio (https://github.com/grpc/grpc/tree/master/src/python/grpcio) which can't be built by conda on windows, but i can run pip and it will make all the necessary work. Is there a way to run pip and take the build artifacts and upload them to a conda custom channel ?

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/conda/dce915e0-85db-4bfe-a84c-c05ef9830aea%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[David Michael Gang]

Ok, thanks for the clarification!

[Chris Barker - NOAA Federal]

On Oct 29, 2017, at 12:48 PM, Michael Sarahan <msar...@gmail.com> wrote:
 if someone built it for Windows and it's on PyPI, then it is possible to build it for conda.

Indeed. And you can actually run pip install in a conda recipe. So if:

pip install some_package 

Works on your platform, it is almost trivial to make a conda package that is essentially identical to the pip-installed one.

 When you see pip used in conda recipes, you'll also see --no-deps as an argument.

Yes!!!

 repackaging Windows wheels is much less troublesome (though still generally frowned on).

Before you use pip in a build recipe, you want to make sure you know what it’s doing.

If it’s installing a binary wheel, that will probably work, but that wheel may not be fully compatible with the rest of your conda environment. That’s why it’s better to have conda build from source.

But Practically Beats Purity, so sometimes a repackaged wheel is a fine option, particularly if you can control what systems it will be used on.

-CHB