[Maya-Python] be - Command-line powered project configurations

[Marcus Ottosson]

Hi,

I’ve been prototyping a new sort of tool in the past week and would like to hear your take on it.

https://github.com/mottosso/be/wiki

It’s available for both Windows, Linux and OSX and is a method of running software in context with a project without having to specify paths, and looks like this.

$ be in spiderman shot01 animation --enter
$ maya
Launching Maya, for Animation in Spiderman, Shot 01..

It works by associating an item in your pipeline - asset, shot, level etc. - with a format string, the typical Python function you get with strings, "my name is {}".format("Marcus"). The format string defines a directory structure, by mapping the arguments given, in the above case spiderman, shot01 and animation to various parts of your design.

shot: {cwd}/{0}/shots/{1}/{2}

Software aliases are defined on a per-project basis in a similar fashion to how templates are defined; a per-project yaml file with one or more aliases pointing either directly or indirectly to the software it relates to, such as

alias:
- maya: start "" "c:/program files/autodesk/maya2016/bin/maya.exe"

The alias is then available upon entering a project.

In addition, I’m also experimenting the the concept of “presets”. A method of encapsulating a full project configuration in a GitHub repository and registering it with another GitHub repository acting as a central database of shared configurations, such that you can build, share and later use curated configurations by others.

$ be new film-project

The above would then start by looking locally, for any existing presets you’ve defined or downloaded earlier, and otherwise start looking through the database. I’ve mocked up a web application visualising the currently available ones here.

• http://be.abstractfactory.io

And there’s instructions for making your own, and registering them for others to enjoy here.

• https://github.com/mottosso/be-presets/wiki

Presets then contain either just the configuration, or can include starter-content, like basic models, rigs, shaders, sound, music, you name it. The idea is for anyone who downloads your preset to get going quickly with a configuration of their choice; such as a preset for “animated-ad” or “action-game” etc.

Have a look and let me know what you think, there’s some thoughts about the future in the issues section, in particular about entering a running project. Being an early alpha/prototype, there is lots of room to grow and for feedback, but templates are quite well-defined and versioned so you can start designing a new project today, or encapsulating the one you use at your company.

Best,
Marcus

​

--

Marcus Ottosson
konstr...@gmail.com

[Roy Nieterau]

Hey Marcus,

This is hitting a sweet spot.
It's nice to read how simple it works plus it's great to see that it works on the first run (just tested with it).

We've had a chat about this outside of this google groups, it was mostly me asking you how be contributes to a pipeline.
After playing around with it I'm starting to see why it could be such a strength when used in production.
I'm wondering if there are other people here who use something similar to set up an artist's environment correctly for each project (and even for the asset they are working on).

Hope this will spark into a nice discussion about environment management in small to big productions. :)
I'm keeping an eye out on this.

Cheers,
Roy

[Justin Israel]

Hey Roy,

At Weta we have an in-house solution, which acts something like a rez with an environment management layer on top. There are a couple goals and concepts that are similar to be. Some of the file-based configuration ideas are similar. And you have the ability to create "product packages" with a number of directives, create dependencies/requirements, and to set up working areas with specific versions of products. Users can then "be in" a given context, which configures the environment context. From that point, users will end up launching pre-determined versions of software, and custom software can rely on the current context when presenting choices such as assets or browsing locations. 

We also are not coupling it to any particular asset management system. It is meant to form the basis of any downstream process that is making a decision about what an appropriate context would be.

Some differences in our solution is that we don't combine the concepts of directory creation and project with this environment management system. Our system is only concerned with resolving environments for a given context. 

I wonder if be could be combined/integrated with rez to provide the full suite that is similar to what we have internally? I think rez has its own "package" format? Maybe be could have a plugin that could allow it to make references to rez packages, and allow you to be into a context that resolves a set of rez dependencies, and then you end up with full environment and package management?

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/1a43a0af-226e-4d85-8f53-66c64751220d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eduardo Grana]

Hey Marcus!

Great stuff, thanks for sharing.

Cheers!

Eduardo

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOA41fYfq5%2BN4HuJ8ahxmFws-LqKs1fKj-H5KkEsa%3D%2B%2BSA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eduardo Graña
www.eduardograna.com.ar

[Marcus Ottosson]

Thanks Roy, for getting a conversation started. :)

Great stuff, thanks for sharing.

You are very welcome, Eduardo!

I wonder if be could be combined/integrated with rez to provide the full suite that is similar to what we have internally?

I actually just got together with a studio the other day who is using rez to discuss how be could potentially supersede their in-house tools and we came to a similar conclusion. I got to have a little closer look at what rez actually solved for them and it seems from first glance that a separation can be made clear enough for be and rez to work together.

• be does project environment, and a little bit of application environment
• rez does application environment, with a little bit of project environment

For example, the dependency resolution handled by rez and ecosystem isn’t something be will do, in the same sense that building and jumping between directories isn’t something rez does.

In this case, be would provide available software, and rez would resolve their dependencies and set up their environment.

In regards to application environment, there seems to be two schools of thought. One, which is what rez does, is to (1) build an environment fit for each software into the current process - for example, you might want maya, nuke and houdini. Once built, each application is available to you and you launch the one you intend to work with next.

The other method is wrappers, which is what I originally had in mind for be before learning about the former. In this case, an environment is (2) built according to a project, user and task - more or less everything except an application. The application environment is then built by the wrapper, which is typically a shell script setting up an environment prior to launching the application.

The top-most benefits I’ve found about each is that (1) puts you in a shell environment identical to the one an application will have once launched whereas (2) eliminates the risk of conflicting environment variables, such as not being able to build an environment for both Maya and Houdini due to both making use of LD_LIBRARY_PATH. Though I can see the perceived value of (1), I can’t quite figure out a practical use case for it, whereas conflicts are very real and difficult to work around. Any input here would be most helpful.

Maybe be could have a plugin that could allow it to make references to rez packages, and allow you to be into a context that resolves a set of rez dependencies, and then you end up with full environment and package management?

I’ll try and experiment with this next week, spontaneously I considered having rez trigger on an “enter” script.

$ be in myproject myasset mytask
Building rez environment..

But then the question is, which applications do you build for?

You could pass extras to the be command.

$ be in myproject myasset mytask --apps maya,nuke,houdini14.5
Building rez environment..

Or maybe you could defer.

$ be in myproject myasset mytask
$ be apps maya,nuke,houdini14.5
Building rez environment..

In which case maybe rez could be used as-is, without any particular integration?

$ be in myproject myasset mytask
$ rez maya,nuke,houdini14.5
Building rez environment..

The latter seems perhaps the most flexible, doesn’t incur any development/maintenance costs on part of be and also doesn’t eliminate existing be application environment support nor the use of wrappers.

Would something like that suffice?

​

[Justin Israel]

On Sun, Jun 28, 2015 at 9:50 PM Marcus Ottosson <konstr...@gmail.com> wrote:

...

The top-most benefits I’ve found about each is that (1) puts you in a shell environment identical to the one an application will have once launched whereas (2) eliminates the risk of conflicting environment variables, such as not being able to build an environment for both Maya and Houdini due to both making use of LD_LIBRARY_PATH. Though I can see the perceived value of (1), I can’t quite figure out a practical use case for it, whereas conflicts are very real and difficult to work around. Any input here would be most helpful.

I've seen both (1) and (2), and I would opt for (2). While you can always combine things like LD_LIBRARY_PATH, the entire environment still has to agree on things and you could have compatibility conflicts. Approach (2) should restrict those dependencies to just that given application. (1) would be easier to do, since you just put everything into that env and away you go. (2) I think implies a bit more work to make sure that you not only bootstrap the application you want to launch, but also include anything it might need, like plugins, or dependencies that will be used by tools that are run within that app.  

Maybe be could have a plugin that could allow it to make references to rez packages, and allow you to be into a context that resolves a set of rez dependencies, and then you end up with full environment and package management?

I’ll try and experiment with this next week, spontaneously I considered having rez trigger on an “enter” script.

$ be in myproject myasset mytask
Building rez environment..

But then the question is, which applications do you build for?

I think you would have to have a registration of packages for a given context. So "myproject myasset mytask" would derive some kind of package list, at least from the project level, and then you just build that environment. 

You could pass extras to the be command.

$ be in myproject myasset mytask --apps maya,nuke,houdini14.5
Building rez environment..

Or maybe you could defer.

$ be in myproject myasset mytask
$ be apps maya,nuke,houdini14.5
Building rez environment..

It would probably be ideal that users would not have to specify versions of things they will use, since that should be dictated by the project configuration (or any location beneath the project that may have overridden the package definitions).  Lets say your studio uses Maya. You may have ProjectA that is deep into production and crunching on Maya2014. Then you have ProjectB just setting up, and they want to test out Maya2016. You don't want to change the Maya version for ProjectA while they are in production. And you also don't want to make people remember to launch Maya2014 on ProjectA and Maya2016 on ProjectB. A user probably just wants to type "maya" and get the right one. Furthermore, maybe on a given show they want to test a new version of software only on a specific sequence.

So if you were able to maintain a registry of versions of software for different locations, then "maya" is just the correct Maya, wherever you run it. 

In which case maybe rez could be used as-is, without any particular integration?

$ be in myproject myasset mytask
$ rez maya,nuke,houdini14.5
Building rez environment..

The latter seems perhaps the most flexible, doesn’t incur any development/maintenance costs on part of be and also doesn’t eliminate existing be application environment support nor the use of wrappers.

Would something like that suffice?

​

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAGir9OAunGD8Vythjw7P_ixDT74CFsAVug%3DW2B4A-vAw%40mail.gmail.com.

[Marcus Ottosson]

Thanks Justin. That all makes sense to me.

I wanted to show some exciting progress!

• Tab completion
• Gists

Gists means you can make a project configuration and host it as a GitHub gist. It can also be registered with the be database and be given a shortname.

• Example gist
• Database

The really cool thing is that tab completion now works and maintains a live connection to the templates you design, which means you can browse projects, and all existing assets and shots and levels and whatnot in your project by hitting TAB.

Let me know what you think!

​

[Marcus Ottosson]

Oh, and the completion works on all platforms with bash, tested with Git Bash, MSYS2 on Windows, and native bash on Ubuntu. The gif above is running MSYS2.

--

Marcus Ottosson
konstr...@gmail.com

[LIJU kunnummal]

Hi Marucss,
We do have similar sort of system, which sets you an environment for the show.
But the thing which is lacking here is the auto-completion part, which is really helpfull when you have large shows and shots to search for.

And it would be great if you can perform any particular action with an app in that environment, such as

$be in spiderman myshot mytask
$be --app maya --action render
$be --app maya --action publish

[Marcus Ottosson]

Hey Liju!

That’s great to hear, I’ve heard many others telling me this too. It would seem as though it’s scratching a common itch.

About your suggestion to run applications via be, this is what wrappers are for, or Rez. Because an application can sometimes require such a heavily specialised environment - with dependencies towards other applications for example - beyond just knowing about show-specific metadata, it’s better suited for a dedicated utility.

In the simplest sense, you can get what you ask through be with a slightly alternative syntax.

$ be in spiderman myshot mytask
$ maya --action render

In which case maya is either exposed to your PATH or generated on the fly with be. maya is in this case relative your project, and could either be a bespoke wrapper script, or something that solves a Maya environment using Rez.

For example, a maya wrapper could look something like this.

import os
import sys
import argparse
import subprocess

parser = argparse.ArgumentParser()
parser.add_argument("--action")

args = parser.parse_args()

if args.action == "render":
  os.environ["PATH"] = "/path/to/vray-plugin"

if args.action == "animation":
  os.environ["PYTHONPATH"] = "/path/to/animation-shelf"

sys.exit(subprocess.call("/path/to/maya"))

$be —app maya —action publish

Hm, not quite sure what to make of this action. What does it mean for Maya to launch given the action “publish”?

@all, would it be possible to post a short shippet of the general layout of the directory structure you are familiar with from work? I’m looking to stress-test be to see whether there are necessary directory structures it may have trouble handling.

For example, something like this would be great.

projects
├── spiderman
│   ├── assets
│   │   ├── peter
│   │   ├── mjay
│   │   │   ├── model
│   │   │   └── rig
│   │   └── harry
│   └── film
│       ├── seq01
│       ├── seq02
│       │   └── 1000
│       └── seq03
└── hulk

​

​

[LIJU kunnummal]

HI Marcuss,

Hm, not quite sure what to make of this action. What does it mean for
Maya to launch given the action “publish”?

 

$be —app maya —action publish

So, similar to any other action publish can be implemented through command line. By saying that, be looks for its config for publish action and invoke the publish tool using the current environment.
so it would be something similar as,

  $be —app maya —action publish

< maya file to publish > can be easily generated through Rez or any packaging tool through the current environment, 

eg: 

/jobs/< job_name >/users/< my_name >/workspace/< seq >/< shot >/< task >/maya/my_maya_file.mb 

< my_name >, < seq >, < shot >, < task > are already made available.

So, by doing this it could be possible to plug any external publish tool to do publish through command line.

[Marcus Ottosson]

But what exactly would become published?

For example, if I work on animation for shot 5000 of The Avengers, and I save my scene as `my_scene.mb`, alongside other files named equally poor but differently, would publishing this shot mean to publish the..

A) ..latest modified one?

B) ..the one with the highest number (if any; possibly version number)?

C) ..all files?

It seems to me that publishing would require at least one more level of specificity, which in this case would be the actual file. One more if also specifying what portion of a file to publish, such as Mesh A, but not Mesh B.

It sounds like a really good and flexible idea that I'd like to implement, but technically I'm having trouble coming up with a non-ambiguous path for it.

[LIJU kunnummal]

But what exactly would become published?

For example, if I work on animation for shot 5000 of The Avengers, and I save my scene as `my_scene.mb`, alongside other files named equally poor but differently, would publishing this shot mean to publish the..

A) ..latest modified one?

B) ..the one with the highest number (if any; possibly version number)?

C) ..all files?

That's upto your pipeline, which file you wanna pick. If you are keeping a naming convention for your files, then you can go for the latest version, or else you have to specify the file name.

It seems to me that publishing would require at least one more level of specificity, which in this case would be the actual file. One more if also specifying what portion of a file to publish, such as Mesh A, but not Mesh B.

In these cases we should need more details about the file. What we used to do is, we used to keep a package info file(a json file) for storing all information related to the file. and allowed artist to publish in two way,
1. internal(wip) - which is not shared among different disciplines, it's just local to them(department).
2. Release - Which is actual publish and shared among different disciplines.

So, the publish tool take this package file in-order to publish which is having a proper naming convention, and it is mandatory to have one in-order to publish your file.
The CLI option to publish should be another way of doing your publish other than from your host application. In that case an artist who finish their work one day and come next day, then he need not to open the host application for doing only a publish. it can be done through CLI is well.


Thanks
Liju K