Proper way to resolve an environment for an embedded python environment?

[Renaud Lessard]

Hi!

I'm questioning myself on how the best way to resolve an environment for a python environment embedded in a DCC application?

ex:

In CentOS 6, the default python is python-2.6.6.

A lot of DCC applications provide their own python version.

For example, maya provided a embedded python-2.7 environment.

This mean that when using rez in the terminal to create a environment were Maya will live, all dependencies that depends on a specific python version (ex:PySide) will resolve to python-2.6.6. 

Is there a preferred way to deal with this situation in REZ?

A workaround I see is to create a "fake" python-2.7 package locally and make the "maya" package depend on it.

However it feel really hacky. Also it could prevent a situation where I'll want to use maya binaries in a python-2.6 environment.

Any insight is appreciated, thanks!

[Fede Naum]

Hi Renaud,

We solved this defining PYTHON_EXE on all packages (python, maya, xsi, houdini, etc)

Later, we always use PYTHON_EXE everywhere to run a program or open a terminal in an environment, that will open the correct python interpreter

in python package PYTHON_EXE=!ROOT!/bin/python

in houdini package PYTHON_EXE=!ROOT!/bin/hython

in maya package PYTHON_EXE=!ROOT!/bin/mayapy

Then, if you have a package that needs to work in all of these you can have variants, and then use $PYTHON_EXEX ie.

name: myApp

variants:

- [ CentOS-6.2, maya-2016 ]

- [ CentOS-6.2, maya-2016 ]

- [ CentOS-6.2, houdini-15 ]

commands:

- MY_APP_EXE=$PYTHON_EXE !ROOT!myApp.py

or if you just need to open a terminal

> rez-env maya

> $PYTHON_EXE

 ( we have also an alias py=$PYTHON_EXE)

>py

Does it make sense?

F

[Edwin Erdmanis]

Hey Renaud,

We recently upgraded our python ecosystem from python-2.6 to python-2.7 on CentOS-6 with both python versions co-existing to allow for transition.

Python-2.7 is installed under /usr/local in this situation I believe.

If you don't put a weak requires ~python-2.7 somewhere in your maya environment, you risk loading variants of binary python packages built against python-2.6, which will not always work. If you only use pure python packages you could potentially ignore this.

Also, unless you install and rez-bind a python-2.7 in your tool chain, you may not be able to rez_pip_install python packages that are binary compatible with maya's python interpreter.

hth,

Edwin

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Allan Johns]

Hey Renaud,

My suggestion: put a weak python requirement in your DCC as Edwin suggests; but also, make your python package DCC aware, so that it can change behavior is the DCC is present in the env. For example:

# in python's package.py

def commands():

    if "maya" in resolve:

        # point at maya's python libs...

    else:

        env.PYTHONPATH.append("{root}/lib")

        env.PATH.append("{root}/bin")

Hth

A

[Renaud Lessard]

Hi guys, thanks for the responses!

I'm currently going with the weak package reference as suggested, it make a lot of sense!

I still had to deploy a "fake" python-2.7 package locally on the CentOS 6 workstations to make it work asap. 

I'll check for deploying python-2.7 on thoses boxes asap so I can get rid of this hack.

Cheers!

[Mark Streatfield]

I would caution against making the Python package aware of DCCs and their internals.  Coupling in this way means that if you introduce a new DCC (or even a new version of an existing one) you have the potential need to update the Python package.  I believe Python should be lower in the dependency hierarchy and therefore not affected by the release of a higher level application.  The 'commands' block of your Python package becomes more complex and over time, harder to maintain.

[Andy Nicholas]

@Mark Yep. That certainly makes a lot sense from a dependency point of view. Is there an alternative strategy you'd suggest?

[Mark Streatfield]

Something along the lines of Fede's reply (that's what we implemented together at Animal Logic).

--
You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/2IWclNTJEk0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.

[Allan Johns]

Mark is right that DCC logic in a python package does introduce some problems.

I had an idea though. Consider:

# python 

def pre_commands():

    env._PYTHON_LIBPATH = "{root}/lib"

def commands():

    env.PYTHONPATH.append(env._PYTHON_LIBPATH)

def post_commands():

    unset("_PYTHON_LIBPATH")

# maya etc

def pre_commands():

    env._PYTHON_LIBPATH = "{root}/libpython"

This approach means that DCCs can override the python package's behavior, without having to update python itself when new DCCs are released.

Has anyone tried this approach?

A

--

You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[Andy Nicholas]

@Allan That's a good idea, although I ran into what may be an issue.

I'm on Windows and I have Python 2.7.6 installed in my C:\Python27 folder.

Softimage uses its own version of python which is 2.7.3 located in C:\Program Files\Autodesk\Softimage 2015\Application\python.

Seeing as I don't have Python 2.7.3 installed natively, it doesn't feel quite right to offer up Softimage's version as a regular Python package. But if I don't, I'm unable to resolve a request for "softimage python" as there's no external package of the correct version to resolve against.

In my rather limited experience of Rez, my first impression is that there are two possible ways around it:

1) Create a package for Python-2.7.3 but specify inside a variant of Python specifically for Softimage. This then goes back to the issue that Mark mentioned where there's an unwanted dependency.

2) Create a package called "python_softimage" which is completely independent of the standard Python package.

My gut feeling is that 2 is the better option as it would potentially get around many of the problems discussed. I suspect some people may feel that this goes against the grain on things that I'm not aware of at this stage. Thoughts?

Otherwise will start looking at going down the route that Mark and Fede suggested.

Cheers,

Andy

[Allan Johns]

The problem you run into with a package like "softimage_python" is that it doesn't really work when you have another python-based package that gets used inside of and outside of softimage. That package has to depend on 'python', since 'softimage_python' has no meaning outside of softimage. So you'll end up stuck with libraries that can only be used in softimage but should be possible to use elsewhere. Of course you could get around that by other packages having [[python], [softimage_python]] variants, but that's pretty nasty.

Mark/Fede's suggestion is fine, but I'm thinking about the case where you may need to actually link against the python library. If not, Mark's suggestion sounds simpler. But if you need a 'true' python dependency, imo I'd go with my approach and have python-2.7.3 simply error out if softimage/other doesn't complete it out (ie by setting _PYTHON_LIBPATH); or, and perhaps better, simply install native python-2.7.3 as well.

A

[Andy Nicholas]

Okay, yep. That makes total sense.

[Fabien Castan]

Hi,

On our side, we have created 2 packages:

- "python": an empty package, just there to track dependencies

- "pythonStandalone": a package with the installation of all python binaries

"python" has 2 variants: 'pythonStandalone' and ''.

In maya we declare: requires = ['python-2.7', '!pythonStandalone']

Then all packages can declare dependencies to the required python version.

If you are outside of maya, it will automatically load the pythonStandalone package and in maya it will rely on the one provided by maya.

Regards,

Fabien

2016-07-06 18:37 GMT+02:00 Andy Nicholas <fxnu...@gmail.com>:

Okay, yep. That makes total sense.

--

[Mark Streatfield]

As those variants in the python package aren't mutually exclusive you may find that, one day, when you least expect it, Rez resolves to the incorrect variant. 

We struggled with this a lot - non-mutually exclusive variants causing our resolves to be unpredictable (or, predictably not what we expected). Normally this was our fault somewhere - a misconfigured dependency or version range - but Rez's almost brute force approach to resolving caught us out. 

It'd  also be worth reading the docs in solver.py to understand the resolution processes and variant selection. Even if non-mutually exclusive variants get you the correct resolve, we also found it can be at the cost of resolve time (mitigated through caching). Rez can go quite deep down a branch before finding the conflict and backing out. 

I would therefore potentially suggest the second variant is !pythonstandalone, instead of ''.  

Mark

You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/2IWclNTJEk0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.

[Fabien Castan]

I would therefore potentially suggest the second variant is !pythonstandalone, instead of ''.  

Oh yes! It's more readable and may remove ambiguities.

Thx

[Fede Naum]

Agree with Mark,

recently, I had one more case where the rez solver almost blew out  trying to resolve a not (all) mutually exclusive  Qt/MayaQt/QT_vfx packages (along with other 400 dep on the solve)

But,  coming back to python, we never had any problem with that implementation I sort of depicted before, it is simple, just one line in each package, no antipackages, and it is really 0 maintenance, we set it up 3 years ago and never had to get back to review it.

Fede

[Andy Nicholas]

@Mark @Fede

Good to know your method is tried and tested. Do either of you have any thoughts on Allan's suggestion? I'm setting all this up at the moment and Allan's method seems to be working well so far.

A

[Allan Johns]

I should have pointed out as Mark did about non-mutually-exclusive variants. He's correct - in fact variants weren't introduced with non-mutual-exclusivity in mind.

It does work though, and the variant to be selected in not ambiguous *if* you know the rules. See the 'variant_select_mode' setting for more details.

I've been thinking for a while now that adding an explicit variant selection syntax would be a good feature to have. For example:

]$ rez-env floob-1.2/maya-2015

This would require floob-1.2, and restrict its variants to those overlapping with maya-2015.

A

--

[Jesse]

Resurrecting this thread as I'm running into many similar problems (mostly compiled python packages)...

What about allowing a package to satisfy the requirements of other packages... acting as a surrogate.

A package definition for maya could have:

name = 'maya'

surrogates = [ 'python2.7', 'ocio-1']

(it's then up to the maya package to implement all commands needed for each of the surrogate packages)

In this case, any package that requests python2.7 and maya, would prune the python2.7 package from the list of resolved packages since its requirement is already handled by another package. Packages defined as surrogates take precedence over proper packages.

Another use case I could see for this is creating a package to wrap 3rd party python packages. Put them all in a single site-packages directory, and add each individual package as a surrogate. I would imagine this would complicate the resolve a bit as there would be no proper package definition.

One last use case would be similar to the previous example.. in a studio transitioning to Rez where everything is initially in one giant repo, it would allow the giant repo to be split into more granular packages before physically splitting it (and abstracting the split from end-users)...

Cheers,

Jesse

[Allan Johns]

RE surrogates:

Yep that idea's been floating for a while, though I would call it "provides" (as in, maya 'provides' python-2.7).

However there is a way to get a very similar effect currently. You could have your DCCs set a special env-var, say PROVIDES_PYTHON, and add a weak reference to ~python-2.7. Then in the python-2.7 package, it would implement only a post_commands(), which would effectively noop if PROVIDES_PYTHON is set. Clunky, but I can't see why this wouldn't work.

Hth

A

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Jesse]

Thanks - that would get the job done but as you said, is a bit clunky.