Distributing or hosting module?

[Fredrik Averpil]

Hi,

I've been meaning to learn how to efficiently serve a python module to all workstations/render blades without actually installing it locally...

So far I've just copied already compiled modules from the site-packages folder from a local installation and then I keep these in folders such as python26_win7_site-packages, python26_linux_site-packages on the server. Not the cleanest and nicest way of dealing with this, I guess. But what's nice is my scripts just need to do something like this:

sys.path.append('/server/share/modules/')

import modulex

So it's easy to make new modules accessible for machines quickly.

My question is; how would you guys deal with this when building from source?

Eggs in separate OS folders?

Is it at all possible to build eggs for Win/Linux/OSX from one and the same operating system?

I'm usually on Windows (7). Should I rather be using a Linux environment for this type of work?

Is there an RTFM to this (and/or building eggs)?

Please point me towards any literature or write-up worth reading as I'm new to making eggs (and building from source for that matter).

To be less abstract, I'm looking to build stalker from source and have it available (although stored on the server) for all machines (Win/Linux).

Thanks in advance for any pointers.

// Fredrik

[Erkan Özgür Yılmaz]

Hi,

I should repeat my reply also here,

I try to put everything to the server.

I place Stalker, Anima, SQLAlchemy (has a compiled query engine --I think-- but I don't care about the Windows workstations are not being able use it, it is already fast), Jinja2, comtypes and other python libraries to server, setup PYTHONPATH to include those paths.

And install PySide, PyQt4, psycopg2 to workstations to all the applications that has an internal Python interpretter (Maya, Nuke, Houdini etc.) plus to the stand alone interpreter (python 2.7).

E.Ozgur Yilmaz
eoyilmaz.blogspot.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWO%2BtdFE34yhwgbWc2n%2BuFqUb_XRD1oOO%3DyVEDcms%2BKGtg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tony Barbieri]

We use a similar technique as Erkan, but we don't rely on putting all of our paths in PYTHONPATH any longer.  Instead, the only path in PYTHONPATH is the location of a sitecustomize.py file.  sitecustomize.py is a magic file that python will import automatically at initialization time.  One trick with it is you can only ever have one.  When python scans it's initial lib paths, if it finds a file named sitecustomize.py it will import and execute it.  

The issue that cropped up from using PYTHONPATH is it will insert all paths found in it to the beginning of the sys.path.  If it's only a single path, then using PYTHONPATH should be fine.  If you're going to have multiple network locations in it, you may want to look at alternatives due to paths found in PYTHONPATH being inserted before the standard python lib paths.  Because the network locations were at the beginning, python would stat the network files system for the standard python modules like os, sys, etc causing unnecessary slowdowns.

Instead what we did is we created a second environment variable, PSYOP_PYTHONPATH, and have logic in our sitecustomize.py file to take the paths present in PSYOP_PYTHONPATH and append them to the sys.path at python startup.  That's the simple version of what we are doing with sitecustomize.py, we also have a bunch of additional code for dealing with project context, per show code, etc.

Of course what Erkan said is correct, I just wanted to point out an alternative if you notice some slowdowns once you've implement it.

Best,

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGNmyx69utr_3DwbbSPgC6Oo%3DTzjgaS%3DcM8mu%2BhGg4MSYtcpag%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
-tony

[Chad Dombrova]

We use pip to install and manage all 3rd party libraries.  

To do this, we create a distutils.cfg on the local workstation of each developer who will be using pip to control the directory layout:

chad$ cat /usr/local/python-2.7.4/lib/python2.7/distutils/distutils.cfg

[install]

install-base=$PYTHON_THIRD_PARTY

install-purelib=$PYTHON_THIRD_PARTY/lib/pure

install-platlib=$PYTHON_THIRD_PARTY/lib/$OS_ARCH

install-scripts=$PYTHON_THIRD_PARTY/bin

install-headers=$PYTHON_THIRD_PARTY/include

install-data=$PYTHON_THIRD_PARTY/data

then we add $PYTHON_THIRD_PARTY/lib/pure and $PYTHON_THIRD_PARTY/lib/$OS_ARCH to the PYTHONPATH and $PYTHON_THIRD_PARTY/bin to the PATH.  

note that the $PYTHON_THIRD_PARTY variable includes a version directory (e.g. 2.6, 2.7) which keeps versions completely independent of each other.  This is particularly important for executable scripts (in the bin dir), because  the path to the interpreter gets baked into the script during install.  when using the default layout, all versions of python share the same bin directory, so if you installed version 1.1 of packageX using python 2.6, then 1.2 of packageX using python 2.7, the executable for packageX in the bin directory will use version 1.2 with python 2.7.  we need more control than that, hence the distutils.cfg file.

Also note that this layout works for us because our python executables are in the same location on osx and linux, and so the interpreter specified on the first line of the scripts (e.g. #!/usr/local/python-2.7.4/bin/python) will exist regardless of which OS the package was installed from.  A safer solution would be to set install-scripts=$PYTHON_THIRD_PARTY/$OS_ARCH/bin

Lastly, we use a package management system to set our environment variables.  we have a custom in-house solution that we use currently, but I've been collaborating with a few individuals from different studios on a next-generation open source solution incorporating the best ideas from different studios:  https://github.com/nerdvegas/rez.  version 2.0 (on the 2.0 branch) is going to be a massive update and a breaking change from 1.0, so you might want to wait a month or so before adopting, but feel free to jump in and let us know what you think.

chad.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsQOUSf54Qr%2BpHVwcBkyRXt47u_W6NsxcxKM%3DTasyYCoeg%40mail.gmail.com.

[Tony Barbieri]

Hey Chad,

Rez looks really interesting.  I remember looking at it awhile ago.  Any chance 2.0 will support Windows?  If not, have the OS specific components been isolated so I could look at getting it to run?  The one silver lining is we are already running bash on Windows...

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7HeY%3D9xmbBLJ8KQL-ZseEcK9CEx7LhvV6FdVeqCsRpcAA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
-tony

[Marcus Ottosson]

Me and Thorsten Kaufman just started work on the Windows version of Rez, it'd be great to have you with us.

https://github.com/nerdvegas/rez/issues/55

https://github.com/instinct-vfx/rez

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsQQ4DuX_Ywzru_YSEeX4D12aqS3E7yTRcQQN-7xnSEsYQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Chad Dombrova]

Rez looks really interesting.  I remember looking at it awhile ago.  Any chance 2.0 will support Windows?

We've removed all the bash requirements and windows support is under way. Here is the master ticket: https://github.com/nerdvegas/rez/issues/55

Here is the fork where some folks are starting to hack on it: https://github.com/instinct-vfx/rez/tree/2.0-windows

chad.

[Justin Israel]

We actually avoid eggs and just deploy everything to a network location right now. No local installs. Then we use the environment management system to sort out the paths for a given env.

Gunna need to check out rez.  I remember coming across the name before. The proposed updates sound cool about moving away from the hard bash requirement and making it a plugin system.
Does it focus on creating bootstrap wrappers for entry point defined by packages? Or does it try to build complete shell environments together for all requested packages in locations? That is,  does it support the concept of being in environment A and launching an application from package B which uses a given version of Qt. And then in the same shell launching an application from package C  which needs a different version of Qt?

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7E6N_y8%3D2xjTynTRtcVuBtN-saK1X%2Bi0ERT5eonqFnt6A%40mail.gmail.com.

[Marcus Ottosson]

Justin, have a look here

https://code.google.com/p/rez-config/downloads/list

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0FVGGEf0hv-Ykf3WOcYZrOBzPSd%2BO3MRc37vqquXbydQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

Thanks Marcus. Will check those out.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODw13Jz25KTRkoFeTXGp_FJWEcASdM_rxaFdMc2vsx7QQ%40mail.gmail.com.

[Fredrik Averpil]

Thanks guys for all your input. How come you don't use eggs?

I thought eggs with egg-infos were supposed to be the easy way to maintain versions and distribute modules?

Fredrik

[Justin Israel]

We manage versions with a custom packaging and deploy system that is integrated with the environment management system. The eggs can be annoying because they require being reliant on either having pth files being updated or having the eggs added to the path as opposed to just a normal package.
I think if you have the versions and packaging managed externally then the eggs are superfluous.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWMxKgHOat0ua%2BO4msn21Xc2BBXpOg46Ytdx-zXi4Bj-XQ%40mail.gmail.com.

[Marcus Ottosson]

My addition to this topic would be a similar approach to Justin’s. Host packages individually and modify PYTHONPATH based on your criteria, such as project. Each project could then define their own paths, pointing to their relevant versions of their relevant packages.

/hulk/scripts/pakA/__init__.py
/hulk/scripts/pakB/1.4.7/pakB/__init__.py
/hulk/scripts/pakB/1.4.8/pakB/__init__.py
/hulk/metadata.json --> {'PYTHONPATH': 'scripts/:scripts/pakB/1.4.7'}

And then, it could also be useful, in cases where you’d like per-shot or per-asset overrides of your PYTHONPATH, to get metadata to cascade.

/hulk/metadata.json
/hulk/shots/1000/metadata.json --> {'PYTHONPATH': '../../scripts/pakB/1.4.8'}

Here, shot 1000 introduces additional dependencies which would be added to the initial /hulk/metadata.json. At this point, you can establish a foundation metadata container at the root of each project (or all projects) and override essentials as you go.

You could roll your own cascade mechanism, or you could have a look at a library I'm working on called openmetadata;

# This would be the gist of it
import openmetadata as om
location = om.Location('/hulk/shots/1000')
pythonpath = om.Entry('PYTHONPATH', parent=location)
om.inherit(pythonpath)
print pythonpath
'/projects/hulk/scripts/:/projects/hulk/scripts/pakB/1.4.8'

Here is a github link and here’s an example on how to set up the metadata for inheritance.

Best,
Marcus

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0jiVCH1hQ8%3DyURDL9vtcOwkkSVkmOsAbpLHseyFVJ_RA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

Hi Marcus,

Thank you for that thorough explanation. You suggested loading packages based on project. I'm thinking I'd be better off here to load packages based on software config. Also, I have a question regarding installing and building packages (how can I know that they are completely standalone?)... 

Loading packages based on software config

Here in our studio, I'm building a software launcher which essentially generates environment variables and launches the appropriate executable. I think that for me it would make more sense to generate the PYTHONPATH based on which combo of selections users are making in the app launcher. For example, they may be choosing to launch Maya 2014 with V-Ray 2.45 (of a certain nightly build) and some other plugin. We don't use environment paths to specify a shot or a sequence, so for as long a user has the app opened, the environment variables generated apply for that app.

What do you guys think about such an approach?

Installing and building packages - are they standalone?

Also, since I'm new to pip and setuptools, I'm just wondering... when I perform a pip install --target //server/share/package... into a certain directory, I've been having troubles confirming that this package is now completely standalone and can be imported from any other machine of the same type without anything but a local python installation. But this seems to be the case. Am I correct?

Also, for some package's source (downloaded from e.g. github/google code), I've been using setuptools to build the project: python setup.py build (avoiding eggs)

Then I manually move the built project into a certain location. Again, I'm assuming the package built is completely standalone. This is right, right?

When "pip installing" or "setuptools building" packages, I'm doing that from my Win7 64-bit machine and from a virtual CentOS 6.4 64-bit machine (with a virtualenv with --no-site-packages). Seems to work nicely. If you have any pointers to give me here, as I'm new to this, I would appreciate that very much.

I'm truly thankful for all your fantastic input, guys. If you end up visiting siggraph this year I definitively owe you a beer or two :)

This mailing list is invaluable!

Regards,

Fredrik

PS. Marcus, what are you using to get such nice code formatting in your emails? :)

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAuk5Y7KmpXkLPZnkpjrXi0ceZqp%3DMuVfswQ1HYv745OQ%40mail.gmail.com.

[Marcus Ottosson]

Thank you for that thorough explanation. You suggested loading packages based on project. I’m thinking I’d be better off here to load packages based on software config.

You’re welcome. :) And yes, that certainly makes sense. How about this:

$ /root/metadata.json
{
    'maya': {
       'PYTHONPATH': '/root/maya/scripts'
    },
    'nuke': {
        'PYTHONPATH': '/root/nuke/scripts'
    }
}

Something you could then cascade, perhaps on a per-project basis, should you need it:

$ /projects/hulk/metadata.json
$ /projects/spiderman/metadata.json
$ /projects/lotr/metadata.json

And viola, you’ve got metadata on a per-project, per software basis.

You could even do:

# Custom environment setup, for a single user on a single shot within a single project
$ /projects/hulk/shots/1000/private/marcus/metadata.json

And finally, for individual versions of software:

$ /root/metadata.json
{
    'maya-2014x64': {
       'PYTHONPATH': '/root/cool/scripts'
    },
    'maya-2009x86': {
        'PYTHONPATH': '/root/less-cool/scripts'
    }
}

However, .json and others can get quite large and or messy when taken to this extreme. Which is another reason I need openmetadata. :)

Hope it helps.

ps. I’m a markdown fanatic and found this plugin for Chrome, http://markdown-here.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWN4PCxu%3DjvME%2BOnPmKUsUh27COdV23Hx%3DHP-0tMJ6rApg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

On May 14, 2014 6:38 PM, "Fredrik Averpil" <fredrik...@gmail.com> wrote:
>
> Hi Marcus,
>
> Thank you for that thorough explanation. You suggested loading packages based on project. I'm thinking I'd be better off here to load packages based on software config. Also, I have a question regarding installing and building packages (how can I know that they are completely standalone?)... 
>
>
> Loading packages based on software config
>
> Here in our studio, I'm building a software launcher which essentially generates environment variables and launches the appropriate executable. I think that for me it would make more sense to generate the PYTHONPATH based on which combo of selections users are making in the app launcher. For example, they may be choosing to launch Maya 2014 with V-Ray 2.45 (of a certain nightly build) and some other plugin. We don't use environment paths to specify a shot or a sequence, so for as long a user has the app opened, the environment variables generated apply for that app.
>
> What do you guys think about such an approach?
>
>

This is actually my ideal situation too. Applications that are bootstrapped to pull in just the environment they and their dependencies need to launch. Two packages that have conflicting dependencies should be able to launch from the same environment.  But it would be a combination of that and the current set location within a project dictating which versions of apps you should be using.
Project A might be using Maya 2013
Project B might be using Maya 2014
And a specific sequence or shot within Project B might want to be testing Maya 2015

> Installing and building packages - are they standalone?
>
> Also, since I'm new to pip and setuptools, I'm just wondering... when I perform a pip install --target //server/share/package... into a certain directory, I've been having troubles confirming that this package is now completely standalone and can be imported from any other machine of the same type without anything but a local python installation. But this seems to be the case. Am I correct?
> Also, for some package's source (downloaded from e.g. github/google code), I've been using setuptools to build the project: python setup.py build (avoiding eggs)
> Then I manually move the built project into a certain location. Again, I'm assuming the package built is completely standalone. This is right, right?
> When "pip installing" or "setuptools building" packages, I'm doing that from my Win7 64-bit machine and from a virtual CentOS 6.4 64-bit machine (with a virtualenv with --no-site-packages). Seems to work nicely. If you have any pointers to give me here, as I'm new to this, I would appreciate that very much.
>
>

Technically someone could do anything they want in their setup.py so I don't think it could be said that it is always standalone. The entry point scripts it can generates can also have specific shebang lines (non Windows users) .
Are you referring to any dependencies that setuptools might pull down? It doesn't package them up within the Build or anything. It just makes sure they are all there and installed to your prefix location. So if you build something that downloads 4 dependencies but you only copy off the original build elsewhere, then you would be missing those dependencies.

> To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWN4PCxu%3DjvME%2BOnPmKUsUh27COdV23Hx%3DHP-0tMJ6rApg%40mail.gmail.com.

[Tony Barbieri]

I highly recommend developing your launcher to take into account the project context.  Even if you aren't technically using that functionality in the beginning, having the users first set their project context then launching the appropriate application with it's dependencies could save you a lot of headaches in the future.  We haven't gone as granular as having sub project contexts such as sequence/shot/asset (at least at launch time, once in the app we do), but we have the ability to set that up in the future if necessary.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0_-wW_X%2BZHqYOzR9Zc9LPDTNcWk09_vHWwfCBVAyrizQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
-tony

[Matt Estela]

You might wanna take a look at rez by Allan Johns.

https://github.com/nerdvegas/rez

He developed it at Dr.D for Happy Feet 2, that show had an insane churn rate of package updates, software releases, different departments requiring different things at different times, even different shots needing python package locked at X while the render was locked at version Y. Rez would get context on what you were trying to do and REZolve (geddit?) a valid set of packages and paths for your needs. You could give it a date and it'd resolve an environment setup for that day (or hour), display reasons or conflicts if things couldn't be resolved, took a tricky problem and made it workable.

It's one key problem at DrD was speed; when having to resolve hundreds of package tree dependencies it could delay for up to 30 seconds before giving you what wanted. Post-show Allan re-wrote the core of it in C++ so now its apparently very fast, and is being adopted in a few studios. Worth a look if package resolution is becoming an issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsT8ZejEMrn%2B%3DribTzPESGu%2BTKFB2U%3DRLZ3wa-REpFvNkQ%40mail.gmail.com.

[Marcus Ottosson]

I made a write-up about context sensitivity in projects/assets/shots recently - with pros and cons and a simple reference implementation, from the point of view of the shell.

Terminal Context Sensitivity

Maybe it could help.

Best,

Marcus

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsT8ZejEMrn%2B%3DribTzPESGu%2BTKFB2U%3DRLZ3wa-REpFvNkQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

Thanks everyone for this invaluable input! :)

// Fredrik

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODGgtRP0Be3nOwHG7FOee7GFYP%2BSK3GaQNOFvHdQnCk-w%40mail.gmail.com.

[Tony Barbieri]

I'm very interested in looking into Rez.  Here at Psyop, we run all 3 OS's (Windows, OSX and CentOS 6.*).  Windows is the predominate OS for artists and workstations so are there any updates on the 2.0 branch and windows support?  Is the windows fork up to date and usable?  Happy to help with any integration issues once I've wrapped my head around how it all flows and works.  I've written a fully cross platform bash context environment so I'm familiar with a few of the gotchas between MSYS, CYGWIN, and the other OS bash environments...

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWM-epTH314hrTymVoOSXD8ZGOZD97Pwwg_BUj00ddupXQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
-tony

[Marcus Ottosson]

That would surely be helpful, Tony. 

At the moment, both me and Thorsten are invested in other projects and mainly awaiting a usable release of Rez 2.0 on any platform (which will be Linux for starters). Once it's up and running there, we can start looking at getting it running on Windows and from what I gather it shouldn't be too long before that happens.

Albeit rather empty at the moment, you're welcome to post any suggested expertise you may want to share here:

https://github.com/instinct-vfx/rez

For instance, your input on which terminal to use under Windows would be a good topic to start with. I was always thinking cmd.exe, despite its limitations, to make Rez as friendly as possible to Windows users. But if you can think of reasons to use Cygwin or anything else, do bring it up and we'll talk about it.

To get up to speed on what it can do, I would suggest having a look at the videos. They're for version 1, but I'm sure the concepts remain throughout the next release.

https://code.google.com/p/rez-config/downloads/list

Best,

Marcus

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsTLDY3B%3D9Ttg%3Dwtz8d0c%3D5-Bugjggx1G9puFfTsdm252A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com