new installation procedure
allan.johns
Our python rez packages add their install location to PYTHONPATH. The problem I get is when I do this:
]$ rez-env python-2.7
...on a Rez install that's using a different python version (2.6 in this case). The rez cli tools are patched so that they're able to work in the target resolved environment. A patched cli tool looks like this (note: simplified)
#!/tools/install/linux_ub12_x86-64/opensource/python/2.6.8/payload/bin/python2.6
import sys
import os.path
import site
_script_dir = os.path.dirname(__file__)
_install_base = os.path.join(_script_dir, '../../..')
_install_base = os.path.realpath(_install_base)
site.addsitedir(_install_base)
sys.path.insert(0, _install_base)
from rez._sys import _forward_script
_forward_script('context')
The import in bold fails, because it ends up loading python-2.7 builtin modules, rather than python-2.6. Fudging around with sys.path gives varying broken results. Now our installs of python might even be nonstandard and contributing to the problem, but that's even besides the point I think, because it still means that other studios will hit the same kinds of problems.
So the question is how we do this properly deal with this? Is a virtualenv bootstrap still an option (Chad I seem to recall last time we talked about this you were going off virtualenv?).
On our previous OS (centOS) I hadn't run into this problem, but that may have just been sheer luck (the same mix of different python version modules may have happened to work). For those interested, here's the stacktrace:
Traceback (most recent call last):
File "/home/ajohns/py/lib/python2.6/site-packages/rez-2.0.ALPHA.1-py2.6.egg/rez/bin/rezolve", line 3, in <module>
import os.path
File "/tools/shed/method.rez/python/2.7.3/platform-linux/arch-x86_64/os-Ubuntu-12.04/libpython/os.py", line 398, in <module>
import UserDict
File "/tools/shed/method.rez/python/2.7.3/platform-linux/arch-x86_64/os-Ubuntu-12.04/libpython/UserDict.py", line 84, in <module>
_abcoll.MutableMapping.register(IterableUserDict)
File "/tools/shed/method.rez/python/2.7.3/platform-linux/arch-x86_64/os-Ubuntu-12.04/libpython/abc.py", line 109, in register
if issubclass(subclass, cls):
File "/tools/shed/method.rez/python/2.7.3/platform-linux/arch-x86_64/os-Ubuntu-12.04/libpython/abc.py", line 184, in __subclasscheck__
cls._abc_negative_cache.add(subclass)
File "/tools/shed/method.rez/python/2.7.3/platform-linux/arch-x86_64/os-Ubuntu-12.04/libpython/_weakrefset.py", line 84, in add
self.data.add(ref(item, self._remove))
TypeError: cannot create weak reference to 'classobj' object
thx
A
Chad Dombrova
i think i see what’s going on here, but i’m not 100% certain.
if my hunch is correct, the fact that our deps are all package into vendor means we probably have an easy way out. i believe you can use something like #! /path/to/python -E to avoid reading the PYTHONPATH variable. next, i would get rid of the site.addsitedir since that is likely to cause problems and just have the patch code chagne sys.path. to do this, you have to install with pip instead of setuptools, since setuptools creates those awful eggs with everything inside of it, and i believe this is what requires the site.addsitedir.
hopefully that gives you something to go on.
chad.
Allan Johns
--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.
Chad Dombrova
Sent from Mailbox
Chad Dombrova
rez setup.py notes:
Summary
requirements
should work via `pip install` or `python setup.py install`
should be as zero-conf as possible
python run from a virtualenv installed on the server is slower (>0.3 seconds). even on fast file server
setuptools errors if install directory does not support .pth file. e.g. when running `python setup.py install --prefix=~/reztest`.
to avoid egg files (and thus .pth requirement) use: python setup.py install --prefix='~/reztest' --record='record.txt' --single-version-externally-managed
however, can't avoid using setuptools: distutils-only install does not work with pip
installing rez arch / platform / python packages:
option 1: install as rez package data (i.e. under the rez python pacakge)
exists just for testing period
only used if no REZ_PACKAGES_PATH
option 2: install as 'data_files' data (not package data)
create a custom builder to create them: https://github.com/quasiyoke/keys_of_peace/blob/master/setup.py
allan's setup.py bypasses setuptools and writes out a second set of shell scripts, with boilerplate sys.path bootstrapping added in: I want to avoid writing a second set of scripts, and to instead use the scripts created by setuptools/distutils to simplify install and generate .exes: but how to insert the sys.path boilerplating?
option 1: override distutils.build_scripts.copy_scripts
big function with no good place to override the header
option 2: override setuptools.command.easy_install.get_script_header
only works for setuptools "entry_points": does not affect scripts built the normal way
using `python setup.py install` creates the wretched "load_entry_point" style scripts
requires setuptools to be importable just to launch script
installing from a wheel generates non-setuptools-dependent scripts:
pip wheel .
pip install wheelhouse/*
is there a one line equivalent to the above?
this is not distutils compatible, will require either pip or wheel for local development
a spot check of a few respected modules -- ipython, pep8, celery, pip -- shows that entry_points + wheels are well adopted
because it is pre-packaged, it may prove difficult to do the customization we want to do during the install process
bdist_wheel is almost definitely not using setuptools.commands.install_scripts, which does the entry_points magic. i think it has its own method for doing this. in both cases, the scripts are generated on install, but for wheels i'm not sure if this can be customized since setup.py no longer exists by the time the wheel is installed, so there is no way to monkeypatch the installer.
option 3: copy boilerplate sys.path setup into scripts (BEST)
primary issues are style and portability:
assumes we know how to find lib from bin dir
duplicated code
rez._sys is kind of a hodge-podge
key cli forwarding function in __init__, should be in cli._utils
_setup.py contains stuff needed for setup only, but it is installed. should be in setup.py if not too big
also contains shell completion files
Desired Install Tree
install-dir/ ← prefix / install-base AND install-data
rez/
2.0.40/
package.yaml ← data (custom built)
python-2.7/
bin/ ← install-scripts
lib/ ← install-platlib
rez/
rez-2.0.PRE_ALPHA.40-py2.7.egg-info/
rezplugins/
arch.yaml ← data (static: may be committed to repo)
platform.yaml ← data (static: may be committed to repo)
python/
2.7.4/
package.yaml ← data (custom built)
Self-Contained alternative (see last problem/solution)
install-dir/ ← prefix / install-base AND install-data
rez/
2.0.40/
package.yaml ← data (custom built)
python-2.7/
bin/ ← install-scripts
lib/ ← install-platlib
rez/
packages/
arch.yaml ← data (static: may be committed to repo)
platform.yaml ← data (static: may be committed to repo)
python/
2.7.4/
package.yaml ← data (custom built)
rez-2.0.PRE_ALPHA.40-py2.7.egg-info/
rezplugins/
Overview of the Plan
current install process:
1. pip install to temp location. this does a lot of work to create a second set of executable scripts which know how to find the rez python library even if it is not on the PYTHONPATH. it also generates the base rez packages (arch, platform, python, and os) in a temp location inside the python package
2. run rez-bootstrap to install rez to your REZ_RELEASE_PACKAGES directory and create rez packages out of rez's python dependencies
new plan:
1. add all dependencies into the rez python package such that it is completely self-contained (e.g. instead of "import yaml", do "from rez.vendor import yaml"),
2. attempt to achieve a direct pip install to the REZ_RELEASE_PACKAGES directory. this should be fully operational and testable.
Problems and Solutions
problem:
currently, setup.py runs some post-install code that creates a second set of executable scripts, which are modified to find the rez python package without the need to setup PYTHONPATH. these are then installed to their final location via rez-bootstrap (the executables created by `pip install` are not copied). the problem is: how do we do customize these scripts and also generate .exes on windows?
solution:
copy-and-paste the boilerplate code and use setuptools entry_points
details:
setuptools supports a feature called entry_points, which allows the automatic generation of executable scripts from a list of python modules and functions in setup.py. this is a cool feature, particularly because it can generate .exe files on windows. I explored many options to auto-insert the boilerplate code, but in the end, it was far too complicated just to avoid copying in some repeated code. Also, if we disallow `easy_install` in favor of `pip`, I believe we can get rid of the `site.addsitedir` parts of the boilerplate, which should keep it very short.
problem:
following from the previous solution, one problem with "entry_points" is that the executable scripts that setuptools generates rely on importing setuptools just to run your script (e.g. "load_entry_point('pep8==1.5.6', 'console_scripts', 'pep8')"). since we'd like to keep rez self-contained, it is important to avoid this extra dependency.
solution:
use wheels instead of eggs.
details:
there is a new distribution format called 'wheel' which is replacing 'eggs'. if you're using pip install, many of the packages that you've installed are likely already coming from wheels. for example: ipython, pip, virtualenv, requests, django, numpy, and even setuptools offer wheels. the biggest difference between wheels and eggs, is that wheels are pre-built, which makes them faster to install, but a bit less flexible. one great thing about them is that they only require setuptools to generate the wheel, and not to install it or run the entry_point scripts.
problem:
following from the previous solution (again), one problem with wheels is, while they're simple to `pip install` from pypi, they complicate the process of installing from source. e.g. when you do `python setup.py install` it uses setuptools, bypassing the wheel install, which produces nasty "load_entry_point" setuptools-dependent scripts.
solution:
only allow pip installations. for most people (non devs) this will be fine.
details:
for devs, the way to install a wheel from local source is this:
pip wheel . # alternately: python setup.py bdist_wheel
pip install wheelhouse/pep8-1.5.6-py2.py3-none-any.whl
I would definitely prefer if it was a single line with `python setup.py install` or `pip install`. I'm still digging in to this to see if there are alternatives, but it seems at the moment that this entire chain of solutions is invalidated if we can't agree to remove `python setup.py install` support, as that will create the "load_entry_point" scripts that rely on setuptools being on the PYTHONPATH
problem:
python enforces its own directory scheme when installing, and rez needs to do some heavy customization to pull off an install in one step through pip.
solution:
use a setup.cfg (if using wheels) or a custom installer (if using setuptools) to change the install locations
details:
a custom installer can make the installed directory tree look like a rez package:
class install_(install):
def initialize_options(self):
ret = install.initialize_options(self)
# set default install paths. this can be overridden via command-line or .cfg file
# but it is not recommended
self.install_purelib = '$base/rez/%s/python-$py_version_short/lib' % version
self.install_platlib = '$base/rez/%s/python-$py_version_short/lib.$PLAT' % version
self.install_headers = '$base/rez/%s/python-$py_version_short/headers' % version
self.install_scripts = '$base/rez/%s/python-$py_version_short/bin' % version
self.install_data = '$base'
return ret
to install:
pip install --install-option='--prefix=/path/to/packages/'
this install class monkeypatch (above) does not work with wheels!
Trial and error with wheels and Custom Install Directories:
1. The following installs the wheel to the target dir (note that ~/ is not expanded) but it does not take any of the install-options into consideration, including setup.cfg. the result is just a flat install directly under ~/reztest:
pip install wheelhouse/rez-2.0.0-py2-none-any.whl --target=/Volumes/newhome/chad/reztest
2. pip install does not respect --prefix or --install-base when installing wheels. The following tries to install to /usr/local/python-2.7.4:
pip install wheelhouse/rez-2.0.0-py2-none-any.whl --install-option='--install-base=/Volumes/newhome/chad/reztest'
3. Not using --install-option or --target will respect both setup.cfg and distutils.cfg (YAY):
pip install wheelhouse/rez-2.0.0-py2-none-any.whl
need to find out if wheel will handle creating rez's package.yaml file, which is above install-purelib and thus not "package_data"
problem:
how to install base packages (arch, platform, python, os). we might be able to install these as package data, but there are issues with that. what if someone has modified arch or platform and they are installing a new version of rez: do the old versions just get deleted / overwritten? will pip try to uninstall the previous version of rez before installing a new one, or will it not find it since it is not on the PYTHONPATH? generally speaking, working with rez pacakges will require a certain amount of user interaction, as well as the ability to merge package data.
solution:
install the absolute minimum required to call rez-env rez (arch, platform, os, python?). place these packages inside the rez python package folder so it can be found relative to the module path. naturally, there's not much that can be done with rez until the end-user has created some packages, so this is just for initial zero-conf testing. they can at least create an environment with rez-env. if we also include python package, they can fire up python and import the rez module. the next step would be to call rez-boostrap to copy these packages to the desired location for REZ_PACKAGES_PATH. when REZ_PACKAGES_PATH variable exists, rez uses this instead of the internal packages path.
allan.johns
It doesn't install to REZ_RELEASE_PACKAGES, that location is where rez packages are supposed to go. The path that you bootstrap Rez into is arbitrary - systems department at a studio will have some preferred location they want to install third party software to, likely this is where they would bootstrap Rez to also.
new plan:
1. add all dependencies into the rez python package such that it is completely self-contained (e.g. instead of "import yaml", do "from rez.vendor import yaml"),
This is already done.
2. attempt to achieve a direct pip install to the REZ_RELEASE_PACKAGES* directory. this should be fully operational and testable.
*To an arbitrary directory (as mentioned above).
Problems and Solutions
problem:
currently, setup.py runs some post-install code that creates a second set of executable scripts, which are modified to find the rez python package without the need to setup PYTHONPATH. these are then installed to their final location via rez-bootstrap (the executables created by `pip install` are not copied). the problem is: how do we do customize these scripts and also generate .exes on windows?
solution:
copy-and-paste the boilerplate code and use setuptools entry_points
details:
setuptools supports a feature called entry_points, which allows the automatic generation of executable scripts from a list of python modules and functions in setup.py. this is a cool feature, particularly because it can generate .exe files on windows. I explored many options to auto-insert the boilerplate code, but in the end, it was far too complicated just to avoid copying in some repeated code. Also, if we disallow `easy_install` in favor of `pip`, I believe we can get rid of the `site.addsitedir` parts of the boilerplate, which should keep it very short.
problem:
following from the previous solution, one problem with "entry_points" is that the executable scripts that setuptools generates rely on importing setuptools just to run your script (e.g. "load_entry_point('pep8==1.5.6', 'console_scripts', 'pep8')"). since we'd like to keep rez self-contained, it is important to avoid this extra dependency.
solution:
use wheels instead of eggs.
details:
there is a new distribution format called 'wheel' which is replacing 'eggs'. if you're using pip install, many of the packages that you've installed are likely already coming from wheels. for example: ipython, pip, virtualenv, requests, django, numpy, and even setuptools offer wheels. the biggest difference between wheels and eggs, is that wheels are pre-built, which makes them faster to install, but a bit less flexible. one great thing about them is that they only require setuptools to generate the wheel, and not to install it or run the entry_point scripts.
problem:
following from the previous solution (again), one problem with wheels is, while they're simple to `pip install` from pypi, they complicate the process of installing from source. e.g. when you do `python setup.py install` it uses setuptools, bypassing the wheel install, which produces nasty "load_entry_point" setuptools-dependent scripts.
solution:
only allow pip installations. for most people (non devs) this will be fine.
details:
for devs, the way to install a wheel from local source is this:
pip wheel . # alternately: python setup.py bdist_wheel
pip install wheelhouse/pep8-1.5.6-py2.py3-none-any.whl
I would definitely prefer if it was a single line with `python setup.py install` or `pip install`. I'm still digging in to this to see if there are alternatives, but it seems at the moment that this entire chain of solutions is invalidated if we can't agree to remove `python setup.py install` support, as that will create the "load_entry_point" scripts that rely on setuptools being on the PYTHONPATH
I actually don't mind if the dev's install process is a little more complicated if that gives us a viable and simple install for standard rez users. In that vein, if removing "python setup.py install" support gets us there then I am not against this either.
Ok I think I just got where you're going with this - you're aiming for rez the install, and rez the rez package, to be one and the same? Hence wanting to install it into REZ_RELEASE_PACKAGES_PATH earlier. Yes?
If it makes things easier, I don't think that making rez available as a rez package on install needs to be a strict requirement. Rez-1 had a "rez-egg-install" tool which made it very easy to install python packages as rez packages, and I don't think it's too much to ask that users have to "rez-egg-install rez" after installation to get rez as a rez package in rez-2 (the name should change though, how about 'rip'?). If anything this makes a clearer distinction between "rez the package management software" and "rez the python module", which isn't a bad thing imo. For example, it should be fine that a user be running Rez-2.1.0, and rez-env to an environment containing a package with a dependency on Rez-2.0.0.
It may even be ok to remove bootstrap packages altogether, if there's an easy enough way (rez-ingest) to create them after installation.
problem:
how to install base packages (arch, platform, python, os). we might be able to install these as package data, but there are issues with that. what if someone has modified arch or platform and they are installing a new version of rez: do the old versions just get deleted / overwritten? will pip try to uninstall the previous version of rez before installing a new one, or will it not find it since it is not on the PYTHONPATH? generally speaking, working with rez pacakges will require a certain amount of user interaction, as well as the ability to merge package data.
solution:
install the absolute minimum required to call rez-env rez (arch, platform, os, python?). place these packages inside the rez python package folder so it can be found relative to the module path. naturally, there's not much that can be done with rez until the end-user has created some packages, so this is just for initial zero-conf testing. they can at least create an environment with rez-env. if we also include python package, they can fire up python and import the rez module. the next step would be to call rez-boostrap to copy these packages to the desired location for REZ_PACKAGES_PATH. when REZ_PACKAGES_PATH variable exists, rez uses this instead of the internal packages path.
See above, perhaps we can remove this from the equation altogether.
allan.johns
Chad Dombrova
Note, there are also comments in the section that google has collapsed under "show trimmed content", in case you miss it.
Chad Dombrova
Ok I think I just got where you're going with this - you're aiming for rez the install, and rez the rez package, to be one and the same? Hence wanting to install it into REZ_RELEASE_PACKAGES_PATH earlier. Yes?
To me they are one and the same. I want rez to run from it’s rez package. I don’t want to have two separate locations for rez.
here’s the directory tree that I’m hoping for:
install-dir/ ← prefix / install-base AND install-data
rez/
2.0.40/
package.yaml ← data (custom built)
python-2.7/
bin/ ← install-scripts
lib/ ← install-platlib
rez/
packages/
arch.yaml ← data (static: may be committed to repo)
platform.yaml ← data (static: may be committed to repo)
python/
2.7.4/
package.yaml ← data (custom built)
rez-2.0.PRE_ALPHA.40-py2.7.egg-info/
rezplugins/
notice that there are yaml files inside the rez python library location. the idea is that these get committed to rez and serve as a set of bare-bones internal packages so that users can get up and running without having to bootstrap any packages (os packages should be there too). trying out rez would be as simple as:
$ pip install rez --prefix=/path/to/install-dir
$ rez env python
> $ python
>>> import rez
I think it should be possible. That’s the goal at least.
chad.
allan.johns
On Wednesday, May 21, 2014 11:15:55 AM UTC-7, Chad Dombrova wrote:
Ok I think I just got where you're going with this - you're aiming for rez the install, and rez the rez package, to be one and the same? Hence wanting to install it into REZ_RELEASE_PACKAGES_PATH earlier. Yes?
To me they are one and the same. I want rez to run from it’s rez package. I don’t want to have two separate locations for rez.
So long as it's possible to resolve a rez-dependent package using a different rez version to the one used to do the resolve. Ie, I should be able to create the following package without concern as to what version of Rez the user is using:
# package.yaml
name: reztool
requires: [ rez-2.0 ]
here’s the directory tree that I’m hoping for:
install-dir/ ← prefix / install-base AND install-data rez/ 2.0.40/ package.yaml ← data (custom built) python-2.7/ bin/ ← install-scripts lib/ ← install-platlib rez/ packages/ arch.yaml ← data (static: may be committed to repo) platform.yaml ← data (static: may be committed to repo) python/ 2.7.4/ package.yaml ← data (custom built) rez-2.0.PRE_ALPHA.40-py2.7.egg-info/ rezplugins/notice that there are yaml files inside the rez python library location. the idea is that these get committed to rez and serve as a set of bare-bones internal packages so that users can get up and running without having to bootstrap any packages (os packages should be there too). trying out rez would be as simple as:
$ pip install rez --prefix=/path/to/install-dir $ rez env python > $ python >>> import rezI think it should be possible. That’s the goal at least.
...and be able to resolve an environment containing that package, even though the rez install we're using to do that is, say, version 2.2.0.
Re the platform/arch packages: So you mean these are multi-package static yaml files with the various systems already listed. Ok that sounds good. At resolve time, the implicit packages do detect the system (for example, "~platform-{platform}" is expanded in rezconfig), but that ends up selecting one of these static packages.
What about the 'os' package though? Many of our packages at Method are os-dependent, and we need to be able to set implicit_packages in the same way that happens for platform and arch (ie have "~os-{os}" expand as expected). This isn't going to work for some distros (consider Phil's issue in his recent post) but that doesn't matter - in that case you either write your own os package if you're going to have os-dependent packages, or you just don't have os-dependent packages, in which case the weak package reference never expands. Even if the provided static os packages list isn't extensive (some major linux distros, osx, windows) it would serve as useful boilerplate for studios that do want to make their own os package.
chad.
allan.johns
"...and be able to resolve an environment containing that package, even though the rez install we're using to do that is, say, version 2.2.0."
allan.johns
How does this allow for installing the same version of Rez for different platforms/OSs, that share the same REZ_PACKAGES_PATH?
This isn't a contrived case, it's how things are structured here at Method.
Chad Dombrova
Wrt installing rez to REZ_PACKAGES_PATH:
How does this allow for installing the same version of Rez for different platforms/OSs, that share the same REZ_PACKAGES_PATH?
This isn't a contrived case, it's how things are structured here at Method.
If I understand you correctly, this is also how Luma would be using rez.
The install structure might look something like this:
install-dir/
rez/
2.0.40/
package.yaml
python-2.6/
bin/
lib/
rez/
rez-2.0.PRE_ALPHA.40-py2.6.egg-info/
rezplugins/
python-2.7/
bin/
lib/
rez/
rez-2.0.PRE_ALPHA.40-py2.7.egg-info/
rezplugins/
2.0.50/
package.yaml
python-2.6/
bin/
lib/
rez/
rez-2.0.PRE_ALPHA.50-py2.6.egg-info/
rezplugins/
python-2.7/
bin/
lib/
rez/
rez-2.0.PRE_ALPHA.50-py2.7.egg-info/
rezplugins/
Basically, you just need to put the appropriate bin directory on the PATH. This would most likely be done by a rez-init.sh/csh/bat script that is sourced when each shell starts. W could put copies of all of them in each variant dir. The exact details of how rez gets on the path I think will vary from studio to studio. some studios might want to make a symlink to /usr/local/bin.
chad.
allan.johns
The only rez variants available are for different python versions, but we most probably need rez to be using a different python interpreter for each of these installs. But according to the layout here, that's not possible - there's only one rez-2.0.50 for python-2.6 available.
allan.johns
Chad Dombrova
What I mean is, let's assume we need rez-2.0.50 on Centos-5.2, Ubuntu-12.04 and osx, all using python-2.6.
The only rez variants available are for different python versions, but we most probably need rez to be using a different python interpreter for each of these installs. But according to the layout here, that's not possible - there's only one rez-2.0.50 for python-2.6 available.
allan.johns
Mark Streatfield
Marcus Ottosson
As an aside, how does Windows like the ‘rez’ command being on %PATH% - won’t it actually be looking for a ‘rez.exe’?
This is true. Windows has an environment variable PATHEXT which contains all extensions used when looking for executables in a shell; by default, only exe is in there, so anything without this extension will not get recognised as a command.
Marcus Ottosson
Allan Johns
Chad Dombrova
little known fact: pip (via distutils) will read installation locations from a setup.cfg that resides in the current directory when running pip install. An example file looks like this:
[install]
install-base=~/reztest
install-purelib=$base/rez/python-$py_version_short/lib
install-platlib=$base/rez/python-$py_version_short/lib.$PLAT
install-headers=$base/rez/python-$py_version_short/headers
install-scripts=$base/rez/python-$py_version_short/bin
install-data=$base
It can read a number of variables and even environment variables. here are the built-in variables from distutils/command/install.py:
py_version = (string.split(sys.version))[0]
(prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
self.config_vars = {'dist_name': self.distribution.get_name(),
'dist_version': self.distribution.get_version(),
'dist_fullname': self.distribution.get_fullname(),
'py_version': py_version,
'py_version_short': py_version[0:3],
'py_version_nodot': py_version[0] + py_version[2],
'sys_prefix': prefix,
'prefix': prefix,
'sys_exec_prefix': exec_prefix,
'exec_prefix': exec_prefix,
'userbase': self.install_userbase,
'usersite': self.install_usersite,
}
self.expand_basedirs()
The confusing thing about setup.cfg is that it is also used by developers when packaging their module to distribute via PyPI. I’m doubtful that there is a way for a package to provide install locations via setup.cfg when using pip install from PyPI since that would be a serious security issue.
so, like allan, my plan has been gravitating toward a simple setup script, however, I’m loathe to write something cumbersome and complex: I’d like to leverage the parts of python’s package system that work. with the above knowledge I think we can write an install script that is as simple as:
1) cd to a temp dir
2) generate a custom setup.cfg defining the desired structure
3) pip install rez
This gives us a little bit of freedom to customize the install based on OS, such as the variant directory that rez is installed to (something that would otherwise be difficult with wheels, since they are pre-built). It also provides a way for users to edit the setup.cfg to customize the install however they want.
Another advantage is that it also works in a very similar fashion when installing from source. you just create the setup.cfg next to rez’s setup.py and then run:
pip wheel . # alternately: python setup.py bdist_wheel
pip install wheelhouse/rez-2.0.0-py2-none-any.whl
you can test this for yourself: dump the text above to a setup.cfg file, and run pip install pep8 from the directory where the file lives. pep8 is packaged as a wheel on PyPI, so this confirms that it works with wheels.
chad.
Chad Dombrova
allan.johns
Chad Dombrova
perhaps your pip or setuptools are out of date?
[11:34] ws-050: /var/tmp/piptest chad$ cat setup.cfg
[install]
install-base=~/reztest
install-purelib=$base/rez/python-$py_version_short/lib
install-platlib=$base/rez/python-$py_version_short/lib.$PLAT
install-headers=$base/rez/python-$py_version_short/headers
install-scripts=$base/rez/python-$py_version_short/bin
install-data=$base
[11:34] ws-050: /var/tmp/piptest chad$ rm -rf ~/reztest
[11:35] ws-050: /var/tmp/piptest chad$ pip install pep8
Downloading/unpacking pep8
Using download cache from /pip/downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2F3.3%2Fp%2Fpep8%2Fpep8-1.5.6-py2.py3-none-any.whl
Installing collected packages: pep8
Successfully installed pep8
Cleaning up...
[11:35] ws-050: /var/tmp/piptest chad$ tree ~/reztest
/Volumes/newhome/chad/reztest
└── rez
└── python-2.7
├── bin
│ └── pep8
└── lib
├── pep8-1.5.6.dist-info
│ ├── DESCRIPTION.rst
│ ├── METADATA
│ ├── RECORD
│ ├── WHEEL
│ ├── entry_points.txt
│ ├── namespace_packages.txt
│ ├── pydist.json
│ └── top_level.txt
├── pep8.py
└── pep8.pyc
5 directories, 11 files
I’m using pip 1.5.4 and setuptools 3.4.4
chad.
allan.johns
pip wheel . # alternately: python setup.py bdist_wheel
pip install wheelhouse/rez-2.0.0-py2-none-any.whlChad Dombrova
Here’s my repro:
[12:05] ws-050: /var/tmp/piptest chad$ git clone https://github.com/jcrocholl/pep8
Cloning into 'pep8'...
remote: Reusing existing pack: 2821, done.
remote: Total 2821 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (2821/2821), 844.48 KiB | 734.00 KiB/s, done.
Resolving deltas: 100% (1593/1593), done.
[12:05] ws-050: /var/tmp/piptest chad$ cd pep8/
[12:05] ws-050: /var/tmp/piptest/pep8 chad$ cp ../setup.cfg ./
[12:05] ws-050: /var/tmp/piptest/pep8 chad$ rm -rf ~/reztest
[12:06] ws-050: /var/tmp/piptest/pep8 chad$ pip wheel .
Unpacking /var/tmp/piptest/pep8
Running setup.py (path:/tmp/pip-qdsPsq-build/setup.py) egg_info for package from file:///var/tmp/piptest/pep8
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'testsuite'
warning: no previously-included files matching '*.pyo' found under directory 'testsuite'
no previously-included directories found matching 'docs/_build'
Building wheels for collected packages: pep8
Running setup.py bdist_wheel for pep8
Destination directory: /var/tmp/piptest/pep8/wheelhouse
Successfully built pep8
Cleaning up...
[12:06] ws-050: /var/tmp/piptest/pep8 chad$ pip install wheelhouse/pep8-1.5.7a0-py2-none-any.whl
Unpacking ./wheelhouse/pep8-1.5.7a0-py2-none-any.whl
Installing collected packages: pep8
Successfully installed pep8
Cleaning up...
[12:06] ws-050: /var/tmp/piptest/pep8 chad$ tree ~/reztest/
/Volumes/newhome/chad/reztest/
└── rez
└── python-2.7
├── bin
│ └── pep8
└── lib
├── pep8-1.5.7a0.dist-info
│ ├── DESCRIPTION.rst
│ ├── METADATA
│ ├── RECORD
│ ├── WHEEL
│ ├── entry_points.txt
│ ├── metadata.json
│ ├── namespace_packages.txt
│ └── top_level.txt
├── pep8.py
└── pep8.pyc
5 directories, 11 files
Chad Dombrova
Oh, one thing to note, pep8 comes with its own setup.cfg, which controls how the wheel is built:
[wheel]
universal = 1
[pep8]
select =
ignore = E226,E24
max_line_length = 79
universal just means it will work with the 2to3 converter so it did not affect anything my example, but technically, where I did cp ../setup.cfg I should have done cat ../setup.cfg >> setup.cfg
chad.
allan.johns
Question: We want to have a custom installer (for non-rez-developers) that does this:
1) cd to a temp dir
2) generate a custom setup.cfg defining the desired structure
3) pip install rez
4) update the installed rezconfig file to set default packages_path config entry to match the install.
In any case - should we provide this installer as a separate package, perhaps 'rez-installer'? I have mixed feelings on this - on the one hand this splits rez across repos, on the other, it seems weird that users would download all of rez just to run an install script that then downloads rez again.
allan.johns
Chad Dombrova
In any case - should we provide this installer as a separate package, perhaps 'rez-installer'? I have mixed feelings on this - on the one hand this splits rez across repos, on the other, it seems weird that users would download all of rez just to run an install script that then downloads rez again.
allan.johns
thx
A
allan.johns
allan.johns
This approach is proving to actually work. I have a standalone installer working: I am able to install rez like so:
]$ python get-rez.py INSTALL_PATH # (get-rez.py can be downloaded on its own).
The result appears to be a zero-dependency rez installation. I've managed to make get-rez.py zero dependency also - pip isn't even needed, as it's embedded inside get-rez.py.
I also have a build.py file, the analogous command for developers, that is working also, but pip and setuptools are needed in this case.
I'm off on holidays over this long weekend but I'm planning on finishing this work off starting Tuesday morning.
thx
A
Chad Dombrova
Sent from Mailbox
Mark Streatfield
allan.johns
Here's what I'm thinking more specifically:
A) if REZ_PACKAGES_PATH or equivalent config key is not set, then it defaults to containing two paths: the path that picks up Rez as a Rez package, and the path that picks up the static packages in Rez's 'packages' subdirectory.
B) if REZ_PACKAGES_PATH or equivalent *is* set, then it is used verbatim. It's up to the user to setup their system packages (probably via rez-ingest).
A side-effect of (B) is that it is possible to set a packages path that doesn't include the rez install. Typically I don't think users would do this, but it does give some more flexibility. For example, a studio may want various Rez installs that go further than being dependent on OS. If they really need this, they could always have multiple Rez installs in different paths, then set REZ_PACKAGES_PATH and PATH to switch to the appropriate Rez install, whilst still providing rez as as rez package. This wouldn't be the recommended approach, but it's there if needed.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.
allan.johns
On Saturday, May 24, 2014 3:14:01 PM UTC-7, Mark Streatfield wrote:
Currently the use of these packages is configured in reconfig, I think as an on/off option. Perhaps we should remove this and instead it should be the default value of the packages path in the config file.
I think we should remove this and have packages_path default to null, giving the behaviour I outlined in the previous message.
In this way there is no if/else logic, the behaviour is based on the configuration, consistent with any other packages path. If you are a new Rez user then we have provided some sensible default configuration. If you are we existing Rez user you'll be replacing reconfig anyway (most likely) so the setting will be dropped.
Yep, I think both approaches achieve this.
This initial set of Rez packages would be part of the Rez repository? Or would they come from a rezpackages repository where they are more generally available? With the latter they are easier to reuse (or reinstall) into your actual released packages folder (the non-bootstrapped one). Rez also then may not need special logic to install them, it could just use the 'Rez install' command.This would have the added benefit of changing the default package_repository_url_path value which is currently Luma's repository. Not saying that's wrong exactly, but perhaps the default shouldn't be studio specific. In my head Rez, Rez plugins, and the default formulae repository would be repos under a Rez organisation on GitHub.
"Rez install" is IMO too far off. There is too much work to do before we could have this work properly. We first need to be able to install new variants into already-installed packages, which suggests (rightly so) that we need to be able to merge package resources together. Where rez-install gets complicated is in two other places. First, how does a package define its requirements? They can't be static - a package that depends specifically on boost-1.43.0 isn't very useful when it would actually work with other version of boost. So package definition files either have to inspect available rez packages somehow, or describe version ranges that then get 'baked' on install (but not too baked - a package may require pyfoo-* but then be baked to only major.minor). There are complications related to this. Secondly there's dependencies - If a package install requires other packages to be installed, how do we determine what those packages are? Other package installers (apt yum etc) have resolving algorithms specifically designed to do this, minimising the number of package installs/updates. This is totally different to the current solver and adding a new solver would be a lot of work.
Related to this - I was thinking of adding a new folder to the project to move 'rez-install' related code into for now: "experimental" perhaps. There is some code in to_delete that I don't think necessarily should be deleted, but the files have nowhere else to live. formulae_manager.py would go there amongst others (probably even the whole source_retriever plugin type). Thoughts?
Mark
Glad to hear it is working out.Next up is determining how to provide the base set of required packages -- arch, platform, os, python. What do you think of my proposal to put an initial set of these, including a slew of known OS packages, inside the rez module dir which would only get used if no REZ_PACKAGES_PATH is set?—
Sent from Mailbox
On Fri, May 23, 2014 at 6:42 PM, allan.johns <nerd...@gmail.com> wrote:
Just an update.--
This approach is proving to actually work. I have a standalone installer working: I am able to install rez like so:
]$ python get-rez.py INSTALL_PATH # (get-rez.py can be downloaded on its own).
The result appears to be a zero-dependency rez installation. I've managed to make get-rez.py zero dependency also - pip isn't even needed, as it's embedded inside get-rez.py.
I also have a build.py file, the analogous command for developers, that is working also, but pip and setuptools are needed in this case.
I'm off on holidays over this long weekend but I'm planning on finishing this work off starting Tuesday morning.
thx
A
On Friday, May 23, 2014 2:41:16 PM UTC-7, allan.johns wrote:Sounds good.
On Friday, May 23, 2014 2:38:12 PM UTC-7, Chad Dombrova wrote:
In any case - should we provide this installer as a separate package, perhaps 'rez-installer'? I have mixed feelings on this - on the one hand this splits rez across repos, on the other, it seems weird that users would download all of rez just to run an install script that then downloads rez again.the way that I see packages handling this most often is they provide a get-foo.py within the repo itself, and they provide a direct download link to that file from github . pip does something similar: http://pip.readthedocs.org/en/latest/installing.html#install-pipchad.
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.
Chad Dombrova
On Saturday, May 24, 2014 3:14:01 PM UTC-7, Mark Streatfield wrote:Currently the use of these packages is configured in reconfig, I think as an on/off option. Perhaps we should remove this and instead it should be the default value of the packages path in the config file.
I think we should remove this and have packages_path default to null, giving the behaviour I outlined in the previous message.
install-dir/ ← prefix / install-base AND install-data
rez/
2.0.40/
package.yaml ← data (custom built)
python-2.7/
bin/ ← install-scripts
lib/ ← install-platlib
rez/
packages/
arch.yaml ← data (static: may be committed to repo)
platform.yaml ← data (static: may be committed to repo)
python/
2.7.4/
package.yaml ← data (custom built)
rez-2.0.PRE_ALPHA.40-py2.7.egg-info/
rezplugins/
Do you mean "install-dir"? If yes, then I agree with that too.
This initial set of Rez packages would be part of the Rez repository? Or would they come from a rezpackages repository where they are more generally available? With the latter they are easier to reuse (or reinstall) into your actual released packages folder (the non-bootstrapped one). Rez also then may not need special logic to install them, it could just use the 'Rez install' command.This would have the added benefit of changing the default package_repository_url_path value which is currently Luma's repository. Not saying that's wrong exactly, but perhaps the default shouldn't be studio specific. In my head Rez, Rez plugins, and the default formulae repository would be repos under a Rez organisation on GitHub.
"Rez install" is IMO too far off.
Related to this - I was thinking of adding a new folder to the project to move 'rez-install' related code into for now: "experimental" perhaps. There is some code in to_delete that I don't think necessarily should be deleted, but the files have nowhere else to live. formulae_manager.py would go there amongst others (probably even the whole source_retriever plugin type). Thoughts?
allan.johns
Yes. Btw this is done now, see nerdvegas/wheel.
This initial set of Rez packages would be part of the Rez repository? Or would they come from a rezpackages repository where they are more generally available? With the latter they are easier to reuse (or reinstall) into your actual released packages folder (the non-bootstrapped one). Rez also then may not need special logic to install them, it could just use the 'Rez install' command.This would have the added benefit of changing the default package_repository_url_path value which is currently Luma's repository. Not saying that's wrong exactly, but perhaps the default shouldn't be studio specific. In my head Rez, Rez plugins, and the default formulae repository would be repos under a Rez organisation on GitHub.
"Rez install" is IMO too far off.agree.Related to this - I was thinking of adding a new folder to the project to move 'rez-install' related code into for now: "experimental" perhaps. There is some code in to_delete that I don't think necessarily should be deleted, but the files have nowhere else to live. formulae_manager.py would go there amongst others (probably even the whole source_retriever plugin type). Thoughts?
I wonder if we shouldn't put it on another branch, like 3.0. It's hard to imagine how we can separate the modules and the executables enough that the average user won't try to use them (and then complain about it not working), while not making it so separated that it isn't a pain for those of us who know what we're doing to use it. See what I mean?
I do. How about I make a '3.0_incubator' branch and dump what we have there currently... then in a single commit at some point in the near future I'll clear out to_delete and everything 3.0-related. Which I think includes:
formulae_manager.py
source_downloader plugin
build_utils.py
chad.