maya.cmds vs python openmaya performance for thousands of objects

[matt estela]

Short version:
Would using the python OpenMaya module give big speed gains for
selecting objects and modifying attributes?

Long version:
We have a python based render pass submission tool at work. At its
core its grabbing whatever geometry is defined by a lighter, and
setting attributes. This sometimes means adding attributes first, then
setting them.

We're getting into the situation where we have _very_ heavy scenes,
with thousands of objects, and if lighters use wildcards, eg
'set:tree*', the setting and tagging of objects gets incredibly slow,
like several hours vs a few minutes without wildcards.

Most of the system is written in maya.cmds, was wondering if anyone
knows if there's performance gains to be had in re-writing the
selecting objects/adding attrs/setting attrs stuff with python API
calls, aka the OpenMaya module? I vaguely recall reading there's some
things which are faster, some which are slower (or indeed impossible,
and can only be done in mel/python), but I can't find any specifics
via google.

There's complications of course, which is why the internal algorithm
has to brute force through every shape and every sub-shape of our
alembic style geo format, but was curious if the basic idea of 'yeah,
manipulating hundreds of objects and their attributes is n times
faster with OpenMaya' is worth pursuing.

-matt

[David Johnson]

I'm not sure about the OpenMaya side of the question. There are huge
gains at component level, but thats the limit of my experience.
Are you really "selecting" the objects? Even with cmds, there would be a
big gain by not having to select the objects.

[matt estela]

Well no, not selecting, but creating lists of objects and iterating through them. 

I remembered the python-in-maya google group minutes after asking here, got an interesting answer and some example code. Seems worth investigating. 

Here's my post and the reply:

https://groups.google.com/d/topic/python_inside_maya/usFLgzJBrpM/discussion

[Ark]

Have you compared that with MEL? I did a script that changes thousands
of values for the fluid container voxels, and MEL iterated through them
in seconds where maya.cmds spent several minutes.

[matt estela]

I haven't, no. Last I looked at the code we're doing a lot of list
comprehensions and string manpulations to make our lives easier, but
I'm gonna have a closer look, see if we're missing anything obvious.

[GregNg]

In my limited experience I have come across the same thing that Sagroth found where iterating through MEL commands proved to be much faster than the same commands in maya.cmds. Haven't tried the same loop in PyMEL yet though.

[matt estela]

Ah, I never updated this list with our findings, sorry.

Turns out its maya path wildcards that make it incredibly slow. I bounced this off the very helpful python_in_maya list, results and example code over there:

http://groups.google.com/group/python_inside_maya/browse_thread/thread/bac14b833241ae93?hl=en-US#

basicaly out of 8000 spheres, searching for '*sphere*' was nearly instant. But as soon as you add paths, eg '/*/*/*sphere*', the time jumps to 30 seconds.

--
 
 

[Sid]

So, judging my that thread, the best thing to do is use the maya API and re to repattern the wildcard search?

Sorry, still a bit confused..

Sid

--
 
 

[matt estela]

Pretty much, though interestingly you get a comparable speed gain by doing the same thing with maya.cmds, and probably pymel too (didn't test that tho). 

Basically: 

-getting flat lists of unconnected nodes is fast (all shapes, all bump nodes, all deformers)

-getting and setting attrs on individual objects is fast

-pythons 're' module is fast

-getting lists via connectivity via wildcards is incredibly slow, ie, don't treat the DAG as a unix filesystem and /*/*/*Shape your way around, it sucks

-if you can rethink your search to first get all objects, then filter your saved list based on attrs of those objects, do that (list conprehensions FTW)

-the API is weird, but I understand it a little better now after this experiment, which is nice. maybe time to write a plugin in pyAPI?

--