rez-pip is ready for testing!

[allan.johns]

Hey all,

A new tool, 'rez-pip', is available for testing in the pip_osx branch. It should work on any platform, though it's been most heavily tested on osx so far.

Thanks to Fabio Piparo for doing the final push. This is a brilliant tool to have - it installs PyPi packages, and their dependencies, as rez packages. I've tested with some reasonably tricky packages - numpy, Flask, Celery, and they've all installed successfully.

Ideally you would have pip and python available as rez packages, and if so, rez-pip will use them. But as a fallback, it will simply use a version of pip available on the system if there is one. Just note that this will mean the system python is used - if you're building something compiled that links against libpython, that might be problematic if you have a rez python package that doesn't match. Either way, rez-pip gives you this info on stdout.

Usage eg:

]$ rez-pip --install Flask

...

4 packages were installed:

  itsdangerous-0.24: /Users/racheljohns/packages/itsdangerous/0.24/package.py (platform-osx/arch-x86_64/os-osx-10.11.5/python-2.7)

  MarkupSafe-0.23: /Users/racheljohns/packages/MarkupSafe/0.23/package.py (platform-osx/arch-x86_64/os-osx-10.11.5/python-2.7)

  Jinja2-2.8: /Users/racheljohns/packages/Jinja2/2.8/package.py (platform-osx/arch-x86_64/os-osx-10.11.5/python-2.7)

  Flask-0.11.1: /Users/racheljohns/packages/Flask/0.11.1/package.py (platform-osx/arch-x86_64/os-osx-10.11.5/python-2.7)

2 packages were already installed:

  click-6.6: /Users/racheljohns/packages/click/6.6/package.py (platform-osx/arch-x86_64/os-osx-10.11.5/python-2.7)

  Werkzeug-0.11.10: /Users/racheljohns/packages/Werkzeug/0.11.10/package.py (platform-osx/arch-x86_64/os-osx-10.11.5/python-2.7)

I'll be merging this to master after some more testing.

Thx 

A

[Thorsten Kaufmann]

Hey Alan,

interesting and good to see. For windows using --install-options and prevent usage of wheels is not really an option though (at least for us). Setting up working compilers etc. is a nightmare and many packages provide wheels for the standard distribution of python.

For discussion you can find my current implementation of rez-pip here: https://github.com/instinct-vfx/rez/blob/windows/src/rez/pip.py

I am also not using custom means to implement dependency installation but instead let pip install everything using --target and then use distlib to gather all dependencies and create packages for them.

Cheers,

Thorsten

[b.f...@toonboxent.com]

Thanks, this saves quite a lot of work.

With "ideally" you mean it is not done yet, right?

Here some additional ideas:
- custom python (mentioned above), but also as variant (e.g. we do have python as variant to maya's python)
- custom gcc for binary plugins (should probably match python in most cases)
- custom name mapper (we do all packages lowercase, some might prefix "py_" or similar to prevent name clashing)
- custom config (e.g. we override the package paths for thirdparty libraries via a config within the package)

Our current solution for python packages is a template that does not install the package yet, but downloads the source and creates a source package.
It does not handle dependencies but at least we have opportunity to adapt to the requirements before releasing.

Cheers,
Blazej

[Allan Johns]

@Thorsten: Totally open to taking further contributions that improve things for Windows users. Keep me posted on this!

@Blazej: There are 'python' and 'pip' packages in the repositories dir, so yes you can currently create python/pip rez packages (incidentally, I plan on moving these out to a separate 'rez-recipes' repo). The rez-pip tool will use them if they're available. On your other points - I agree with all of your points. In particular, custom python variant/gcc I think would be supported by providing further package requests as rez-pip args, that are used in the resolved env containing pip.

For example:

]$ rez-pip --install Flask --extra-packages gcc-4.1.2

Cheers

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Thorsten Kaufmann]

Okay, so i have gone through a lot of packages over the weekend and i am all rage face. Let me vent first.

<rant> I hate pip. I hate packaging in python. I hate setuptools. I hate it all. With a passion. Like so much. </rant>

Sorry, that needed to get out, i tried to keep it civilized.

So it turns out, that no matter what variant of targeted installation you use and no matter how hard you try to catch all exceptions there is a lot of stupid issues left. The latest one that i am not able to overcome when using --target is, that scripts are ignored by pip. So only the purelib folder is beeing installed all other data is ignored *brmpf* I did get it to work to some extent by using a custom hacked version of pip to include scripts, but then you can not really distinguish between scripts (need to be on PATH) and the lib (needs to be on PYTHONPATH) etc.

You could get around this by using "--installl-settings" instead, but that disables all wheels and only works for source packages which is kind of a no-go for windows.

So after a lot of hating and mumbling i just tried a different approach and am wondering what you guys think and if this is an option on linux at all.

At first this seemed overkill, but in a proof of concept it solved all my issues with the other approaches so far.

In a nutshell:

• Instead of using pip directly first create a virtualenv in a temp folder
• Activate the virtualenv
• Use distlib to parse the site-packages folder
• Ignore the following packages: wheels, pip, setuptools
• Use get_installed_files to gather all installed files
• Assemble package and release/install

The main difference is that i can actually get all files installed and put them in different folders in my package. Also this removes the need for any --install-options or --target hackery which is shakey at best from my experience. And should generally work with pretty much any package that is pip-able.

As i am not really using linux i was wondering if anyone had any idea if this is feasible there and i am also interested in any feedback. For now i think i will implement this as a seperate tool rather than put it in rez-pip because for one, there is an "official" rez-pip now and on the other hand i want to use external dependencies rather then the bundled ones in "vendor" to get the latest versions etc.

Cheers,

Thorsten

[b.f...@toonboxent.com]

I agree and disagree to: https://glyph.twistedmatrix.com/2016/08/python-packaging.html
It works only if you play by its rules.

Which package caused so much trouble?

Here on Linux I did all the installs manually (setup.py) and was kind of lucky with it.

On windows I just wrapped the (multiverse of) installers so they would extract to the right package.
For this often I used MacGyver's knife: 7zip. It pretty much can extract most installer packages.
Sometimes I just cheated, too by installing and copying but with 7zip this was rarely needed.

So no pip for me. I found wheels which are told to be godsend for pythoneers were most difficult to work with.

I had more issues getting python running without an install as it seems to always require the registry entries (I gave up).
It is fairly beautiful to extract from an MSI via msiexec /a pythonXX.amd64.msi /qb TARGETDIR=${prefix}
It even installs the python.dll next to the executable instead of to system. But it will still not run on other machines from that place.

And then again running executable from a smb share seems to cause calling the pentagon on windows.
Also I never understood why python, even virutalenv relies on shebang instead of simply picking up the first python in PATH (which would be much easier to control).

Yeah, buy a good bottle. I'm glad windows is the nephews stepson in our company.

I wish we'd share rez configures industry wide. I doubt our companies want to compete on that level ...

[Thorsten Kaufmann]

Well, packaging is okayish these days if you are packaging something. It is still a mighty nightmare if you are trying to use all kinds of packages out of your control.

So while i could try and write wrappers for all kinds of installers that would mean that i need everything to have installers. Which simply is not the case. And the whole

point of rez-pip is to be able to install any package available on the index via a simple command. We run an internal devpi server in front of the official index to inject

packages etc. and i also want to make use of that. Also, for some packages we need to build custom versions for use inside DCCs due to pretty much all DCCs using

different runtimes than the official python distribution. We have this available on the devpi and i can just use them from there when using pip.

- Manual installs on windows are a serious pain and break often and easily

- pip just has so many oddities when you are not plain installing from an interpreter but want to go to a target directory. All variants i looked at have bugs, inconsistencies and/or big caveats

We are actually running python from a network share. We are having performance issues still with both python itself, but even more so pulling packages from our storage server but we may

just have to bite it and get a specialised small storage for distributing tools as our main storage is just not optimised for that.

I am all for sharing rez setups. The main issue is probably similar to custom cmake setups in that they tend to rely on other stuff that can not be shared and ymmv when used elsewhere.

Still would be great to have a repository of standard packages and different approaches to packaging with rez!

Cheers,

Thorsten

[Allan Johns]

On that note I've started a repo for shared 'recipes' (have to have a name for rez-based 3rd party installers don't we?):

https://github.com/nerdvegas/rez-recipes

However, I need to add a few features in order for this to be more useful:

1) We need a way for a package to define requirements at build time that are made concrete on release. For example, I may have a package that builds fine against any modern boost version, but needs a specific boost version at runtime, since it needs to link to the lib it was built against. Something perhaps like so:

# package.py

requires = [

  "boost-1.*.*"

]

Let's say here that the '*'s are expanded at build time, and the installed package would then have:

requires = [

  "boost-1.55.0"

]

This feature is needed else we just have a list of recipes that only work when used with quite specific versions of other packages, which doesn't actually translate to the packages' actual build requirements.

2) rez-install.

A tool that is able to build and release a recipe, and any dependent recipes. Basically something more like a proper installer. I can see a fairly simple and elegant way to do this: all we need to do is structure rez-recipes to mirror an actual package repository - ie, named directories with version dirs underneath, and package.py within each. Rez-install would then treat this as if it's an actual package repo, and run a resolve to determine the dependencies of any given package. Then, it would iterate over those package sources, from the bottom of the dependency tree up, and build/release each one.

I suspect that to make this flexible enough for many studios to use without a lot of adjustment, we will need more features. We will probably make use of a feature that is not well known in rez - I'd have to check whether it's still working, but you can implement any package attribute as a function (like commands()), and its return value is made concrete in the installed package (in theory you could actually use this to implement (1) already, but it'd be a fairly verbose way to do it). I daresay it would be worth extending on this and being able to have package attributes that can also *stay* as functions (except some specific attributes, like name and version).

The biggest limit to getting all of this tasty stuff done is my own limited time. I'm actually moving countries pretty soon (LA -> Sydney), so be warned as I may largely drop off the map for a number of weeks during real life teardown/setup.

A

--

You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Thorsten Kaufmann]

rezipes! I like that a lot! And i really like the ideas you posted! That does not stop os from sharing existing setups if they contain all dependent packages though no? To get things a bit headstarted maybe.

Woah @ LA to Sydney. That sounds like quite an adventure. All the best!

Cheers,

Thorsten

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.