Installing pip modules with rez

[Christopher Martin]

Hi,

I'm doing a little research and looking into rez as a way for us to better manage our studio's software environement and one thing that immediately has me curios is - what's the best way to install pip-based python modules with rez? I was digging through the group and it seems that there used to be a rez-egg-install and mention of creating an updated version to use pip, but I can't find any more info on the subject.

Cheers,

Chris.

[Allan Johns]

Hey Chris,

This is entirely possible, I would very much like to add this to rez-2 but have not had the time. Method have an in-house equivalent of this, and it does work. It's extremely handy to install pip-compliant packages as rez packages with a single command.

It's a little tricky to do right - one thing you need to take into account is that the pip used to build the source will need to match the python version your rez package will end up depending on (this might matter - it may have compiled code). Basically this means you're going to need a rez 'pip' package, and use that for the build. So a good prerequisite task might be to add a 'pip' installer package to the new 'repository' directory of the project.

In terms of what to do now, it's fairly straightforward to simply assemble the rez package by hand. Just look at another rez package, then create all the relevant directories and package.py, and copy the built source of the pip package into it. You generally do this under ~/packages first, test that it works and then copy it into your main package release path.

Hth

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Chad Dombrova]

Hi Chris and Allan,

I've been thinking about this lately in the very broadest of terms since Luma will need to have this working again as well.  I think it would be cool to implement this as a new build_system plugin.  Chris, in case you're new to rez's plugins, when rez-build/release is run, a build system plugin detects if it is applicable based on the current directory.  For example, if the "cmake" plugin finds a CMakeLists.txt file then it offers itself as a valid build system.  Likewise a "pip" build system could detect the presence of a setup.py or a requirements.txt file.  That way you can simply navigate to the directory with the setup.py file and run rez-build --install.

Internet access is a problem here as well due to security, so it might need to work in conjunction with a "pypi" source_retriever plugin, so that downloads are properly cached and make available to other users of the pip build system.

[Allan Johns]

Hey Chad,

I think that makes a lot of sense. It deals with the pip issue nicely too (ie, which pip specifically should be used to do the build?) - it's just up to the user to specify the pip rez package they want to use in build_requires. We could also provide a pip installer package to get people started easily (ie under the repository dir).

Speaking of the repository dir, I think it'll make sense to move this out of the rez repo. I'll do this at some point.

A

[Jack Straw]

Hi,

I have just written a bind module for pure python packages;

The bind itself - https://github.com/saddingtonbaynes/rez/commit/a1538d4c946535a76658fbac8f49a21a49673c38

New versions of pip and setuptools to get access to the newer setuptools features - https://github.com/saddingtonbaynes/rez/commit/3f7c289397abb0ef91fc83f19988cfcfd16b9ea4

I haven’t created a pull-request yet as it is working most of the time but I am getting an argument parser error in some seemingly random cases that has me stumped.

So this bind works; rez bind pypackage --pkg="Clique" and most packages I have bound.

But this doesn’t; rez bind pypackage --pkg="termcolor"

It throws; error: invalid command 'bind' Would someone be able to see if this is a issue on other platforms?

I am using python 2.7.8 on windows 7.

Thanks,

Jack.

[Christopher Martin]

Thanks Chad. I was looking into the build plugins a bit but settled on simply calling the pip --install in the CMakeLists.txt itself. A new pip build plugin does sound like a better alternative though.

I'm curious as to how you would envision dealing with any python module dependencies in the requirements.txt? I wanted to maintain each python package installed as it's own rez package so I was doing that for all dependencies, running the pip --install with --no-deps. 

Cheers,

Chris   

[Jack Straw]

I fixed the issue I was having with "termcolor", I needed to monkey patch distutils.core.setup as well as setuptools.setup.

[Allan Johns]

I think --no-deps is the only option actually... by definition rez-build is supposed to build the current package, not any other.

I'd imagine we then have a separate tool, rez-pip-install, that uses this new builder but also deals with the dependency issues, so that you can build and install a setup.py package and it's dependencies, if necessary, in a single step.

[Chad Dombrova]

I'm a little out of date on rez right now, but I remember seeing something newish about installing a package into another package.  Could that be used here?

-chad

[Allan Johns]

No, perhaps you're thinking of being able to release multiple packages from a single vcs repo? That's the closest I can think of.

A

[Chad Dombrova]

That sounds pretty close to what we need. Can it be pushed a little to cover this case?

-chad

[Christopher Martin]

Is the multiple packages from a single repo available currently?

You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/6JlACl5rG_o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.

[Allan Johns]

Yes it's available currently, but perhaps there's some misunderstanding. This just allows for multiple, unrelated packages to be released from the same svn/git/hg repo. This is not subpackages, where one package, with a package.py at its root, may have further packages with their own package.pys within that. This is just side-by-side packages within a repo.

A

[allan.johns]

Hey all,

As another step towards pip-based rez installs, I have done the following:

* added setuptools to the repository of installer packages (so you can rez-build setuptools);
* added pip to the repository of installer packages (so you can rez-build pip);
* added a new rez cmake macro, "rez_pip_install".

This means you can now write a minimal CMakeLists.txt to install a pip package, it looks like so:


CMAKE_MINIMUM_REQUIRED(VERSION 2.8)

include(RezBuild)
include(RezPipInstall)

set(url "/sw/install/...}/subprocess32-${version}.tar.gz")

rez_pip_install(
    subprocess32
    URL ${url}
)

Please note that it uses --no-dep, so dependencies are not installed.

Thx
A

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/6JlACl5rG_o/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rez-config+unsubscribe@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Seth Lippman]

Allan,
  I've been using this quite a bit, works great! I believe you forgot to include this line in your CMakeLists.txt example:

set(version $ENV{REZ_BUILD_PROJECT_VERSION})

Before I added that, my build was failing on this line:

set(url "/sw/install/...}/subprocess32-${version}.tar.gz")

Also on Mac OS X, I was having a bit of trouble until I ran into this thread: (It addresses an issue that can cause rez_pip_install to fail if you are using HomeBrew)

http://stackoverflow.com/questions/24257803/distutilsoptionerror-must-supply-either-home-or-prefix-exec-prefix-not-both

HTH!

Seth