api vs maya.cmds performance re modifying thousands of objects attrs

[matt]

Apologies for the crosspost for anyone on maya_he3d, only remembered this group existed seconds after I posted over there... Have tried to edit and re-word for you smart people. :)

Short version:
Would using the python OpenMaya module give big speed gains for selecting thousands of objects and modifying their attributes?

Long version:
We have a python based, text based render pass submission tool for lighters at work. One of its core functions is grabbing whatever geometry is defined by a lighter, and setting attributes. This sometimes means adding attributes first (or connecting our custom attribute node), then setting them.

We're getting into the situation where we have _very_ heavy scenes, with thousands of objects. Normally our system will process these scenes within a minute or two per frame, if lighters use wildcards, eg 'set:tree*', that can jump to maybe 6 mins per frame. Not too bad.

Things get messy with our alembic style heirachical geo format. It can contain many sub-objects, which our maya plugin doesn't allow us to list or search for sub-objects names easily. Thus, if a lighter wildcards to the sub-object level, eg "set:*|leaves*", the only safe way to do that is to process every object, then every sub-object, unsetting atts those objects which AREN'T leaves, and setting attrs on those objects which ARE leaves. When this happens, processing jumps to 3 hours per frame. Yuck!

In the short term we're getting lighters to be more careful with wildcards, in the mid term getting assets collapsed down so they're not so name and sub-object heavy, and in the long term getting our geo plugin more inspectable. So that's good.

I was curious though.... could this be helped in the short(ish) term by re-writing that part of the code with the OpenMaya module? I recall reading there's many things which are faster, a few which are slower, and fewer still which have no OpenMaya equivalent and can only be done in mel/python. I have a sneaking suspicion one of those slow things was something fundamental like selection, but I'm hoping I'm wrong.

Was curious if the basic idea of 'yeah, manipulating hundreds of objects and their attributes is N times faster with OpenMaya' is worth pursuing. 

-matt

[Justin Israel]

Some of the big speed increases are using the iterators and not having to do a bunch of string operations on dag paths. And the math speedups from using the OpenMaya objects with operators.

Your best bet it to just profile some small tests. You can easily make use of the python `timeit` module to check the difference in speed of operations.

--
view archives: http://groups.google.com/group/python_inside_maya
change your subscription settings: http://groups.google.com/group/python_inside_maya/subscribe

[Justin Israel]

As a random example...I created a nurbsSphere, and just simulated looping over 5000 objects and setting and attrib.

import maya.OpenMaya as om

import time

import maya.cmds as cmds

sel = om.MSelectionList()

om.MGlobal.getActiveSelectionList(sel)

iter = om.MItSelectionList(sel)

obj = om.MObject()

depFn = om.MFnDependencyNode()

start = time.time()

for i in xrange(5000):

    iter.getDependNode(obj)

    depFn.setObject(obj)

    depFn.findPlug("tx").setInt(4)

end = time.time()-start

print end

# 0.0979061126709 seconds

start = time.time()

for i in xrange(5000):

    name = "nurbsSphere1"

    cmds.setAttr("{0}.tx".format(name), 4)

end = time.time()-start

print end

# 0.261173009872 seconds

[Matt Estela]

Wow, definitely seems worth investigating. Thanks for the code snippet and timing info!

[matt]

Hmm... did some experimenting. Using your example as a base, I compared modifying the castsShadows attr on 8000 spheres. I have them grouped in the following way:

`-- set

    |-- a

    |   |-- nurbsSphere0001

    |   |-- ...

    |   `-- nurbsSphere4000

    `-- b

        |-- nurbsSphere4001

        |-- ...

        `-- nurbsSphere8000 

I get very similar results for both api and maya.cmds. Interestingly, I get an incredible slowdown depending on how specific/general I am with the search:

search = set|*|*|nurbsSphereShape*

api = 27.6180000305

cmds = 27.018999815

vs

search = nurbsSphereShape*

api = 0.956000089645

cmds = 0.403000116348

This is my first few hours playing with openmaya, already made some silly mistakes (defining function-sets inside the loop is waaaay slower than outside the loop), but wondering if there's something else I'm missing... would appear wildcards should just be avoided at all costs.  Here's my contrived example:

import maya.OpenMaya as om

import maya.cmds as cmds

import time

#search = "set|*|*|nurbsSphereShape*"

search = "nurbsSphereShape*"

print ("search = %s" % search )

# API based

start = time.time()

sel = om.MSelectionList()

sel.add( search )

iter = om.MItSelectionList(sel)

depFn = om.MFnDependencyNode()

mObj = om.MObject()

while not iter.isDone():

    iter.getDependNode( mObj )

    depFn.setObject(mObj)

    depFn.findPlug("castsShadows").setBool(True)

    iter.next()

end = time.time()-start

print ('api = %s' % end)

# maya.cmds based

start = time.time()

sel = cmds.ls( search )

for each in sel:

    cmds.setAttr("{0}.castsShadows".format(each), 1)

end = time.time()-start

print ('cmds = %s' % end)

[Justin Israel]

Ya, in some cases you can't beat the python commands module if you are only doing a single command. Most of the work is happening behind the scenes in C++. The wildcard searches just appear to be beasty no matter what.

But considering you didn't need a wildcard pattern, and instead just want to say "Apply to all nurbsSurface objects under this root:

#

# cmds

#

start = time.time()

sel =cmds.listRelatives('|set', ad=True, type="nurbsSurface")

for each in sel:

    cmds.setAttr("{0}.castsShadows".format(each), 1)

end = time.time()-start

print ('cmds = %s' % end)

# ** cmds = 0.290652990341 **

#

# api

#

start = time.time()

sel = om.MSelectionList()

dagFn = om.MFnDagNode()

mObj = om.MObject()

dagIt = om.MItDag()

sel.add("|set")

sel.getDependNode(0, mObj)

dagIt.traverseUnderWorld(True)

dagIt.reset(mObj, dagIt.kDepthFirst, om.MFn.kNurbsSurface)

while not dagIt.isDone():

    curr = dagIt.currentItem()

    dagFn.setObject(curr)

    dagFn.findPlug("castsShadows").setBool(False)

    dagIt.next()

end = time.time()-start

print ('api = %s' % end)

# ** api = 0.117326021194 **

[Matt Estela]

(cc my reply to the group)

On Mon, Aug 6, 2012 at 9:16 AM, Matt Estela <matt....@gmail.com> wrote:

Yeah, again it was a contrived example, in production lighters will definitely be using whatever bizarro wildcards they can muster.

As you say it appears to be a core limitation of wildcards, will have to rethink how we let lighters define object selections. In this case maybe we just can't let lighters use wildcards, instead they'll have to pre-define it using sets. Or possibly pre-filtering to specific object types, and running list comprehensions on that.

Hmm, houdini's smart bundles would come in handy here... (dynamic sets based on wildcards, they run surprisingly fast)

Thanks again for the help Justin, you saved me several days worth of research. :)

[Justin Israel]

This is an interesting approach:

start = time.time()

cmds.select('nurbsSphereShape*')

size = len(cmds.ls(sl=True))

cmds.setAttr(".castsShadows", *(1 for _ in xrange(size)))

end = time.time()-start

print end

# 0.243406057358

... Though an extra slow down happens if you have to deselect.

[Justin Israel]

Ah, I figured it out. The 'add' method using a wildcard on a MSelectionList is super slow. If you use MGlobal to fill the list for you, it apparently does it more efficiently:

start = time.time()

search = "nurbsSphereShape*"

sel = om.MSelectionList()

om.MGlobal.getSelectionListByName(search, sel)

iter = om.MItSelectionList(sel)

depFn = om.MFnDependencyNode()

mObj = om.MObject()

while not iter.isDone():

    iter.getDependNode( mObj )

    depFn.setObject(mObj)

    depFn.findPlug("castsShadows").setBool(True)

    iter.next()

    

end = time.time()-start

print ('api = %s' % end)

# api = 0.212560892105

[Marco D'Ambros]

the first question is:

did you use the api 2.0 or the "normal one"? 

the new api suppose to be faster ( almost 3 times more ). 

--------------------------------
Marco D'Ambros
phone : (+61) (0) 435809628

web    : www.marcodambros.com
mail    : dambro...@gmail.com

[Matt Estela]

We're locked to maya2011, API2.0 is a 2012 thing right?

--

[Matt Estela]

Huh, cool!

Back at work today, will try this out...

[Justin Israel]

Yea, it was introduced after Maya 2012. Marco is right though, it is even faster:

import maya.api.OpenMaya as om2

start = time.time()

search = "nurbsSphereShape*"

sel = om2.MGlobal.getSelectionListByName(search)

depFn = om2.MFnDependencyNode()

for i in xrange(sel.length()):

    mObj = sel.getDependNode(i)

    depNode = depFn.setObject(mObj)

    depNode.findPlug("castsShadows", True).setBool(True)

    

end = time.time()-start

print ('api2.0 = %s' % end)

# api2.0 = 0.123915910721

[Matt Estela]

Damnit. :/

--

[Justin Israel]

I've been playing with cython lately on another project, and I decided to see what would happen if I did some Maya operations with it. 

I made a really contrived example of wildcard selecting over 5000 spheres and doing some string filtering to the results before returning a list. Interestingly, cython gave at least a 35% speedup over the Maya Python API 2.0 approach.

Makes me think that a lot of more specialized operations could be heavily sped up that way by moving them over into a C++ environment of execution.

[Matt Estela]

I had another play with this too. :)

Getting maya to do the wildcard matching still takes about 27 seconds
with the new method.

Was curious how slow/fast it'd be if I did the wildcard match myself.
I get all the shapes, get each shapes dagPath, test it with the re
module, and run my setAttr if it matches. Doing this, the time comes
down to 0.5 of a second. Tried a few runs with matching half the
shapes, all the shapes, and no shapes, performance beats the builtin
wildcard by a huge margin.

search = nurbsSphereShape*
pattern = |set|*|*|nurbsSphereShape*
api = 0.460000038147 sec, 8016 shapes modified
cmds = 26.9750001431 sec, 8016 shapes modified

search = nurbsSphereShape*
pattern = |set|a|*|nurbsSphereShape*
repattern = [|]set[|]a[|].*[|]nurbsSphereShape.*
api = 0.319000005722 sec, 4323 shapes modified
cmds = 14.5989999771 sec, 4323 shapes modified

search = nurbsSphereShape*
pattern = |set|*|*|badmatch*
api = 0.182000160217 sec, 0 shapes modified
cmds = 11.1970000267 sec, 0 shapes modified

here's my fugly test code:

import maya.OpenMaya as om
import maya.cmds as cmds
import time

import re

search = "nurbsSphereShape*"
pattern = "|set|*|*|badmatch*"

# change the maya pattern to a re friendly pattern; | becomes [|], and
* becomes .*
repattern = re.sub('[|]',r'[|]',pattern)
repattern = re.sub('\*',r'.*',repattern)

print ("search = %s" % search )

print ("pattern = %s" % pattern )
print ("repattern = %s" % repattern)

# API based

start = time.time()
sel = om.MSelectionList()

om.MGlobal.getSelectionListByName(search, sel)
iter = om.MItSelectionList(sel)
depFn = om.MFnDependencyNode()
mObj = om.MObject()

dPath = om.MDagPath()


count = 0

while not iter.isDone():
iter.getDependNode( mObj )
depFn.setObject(mObj)

om.MDagPath.getAPathTo(mObj, dPath)
path = dPath.fullPathName()
if ( re.search(repattern, path)):
depFn.findPlug("castsShadows").setBool(True)
count = count + 1
iter.next()
end = time.time()-start
print ('api = %s sec, %s shapes modified' % (end, count))

# maya.cmds based
start = time.time()
sel = cmds.ls( pattern )
count = len(sel)

for each in sel:
cmds.setAttr("{0}.castsShadows".format(each), 1)
end = time.time()-start

print ('cmds = %s sec, %s shapes modified' % (end, count))

[Matt Estela]

On closer inspection, same technique holds true with regular maya.cmds() too. Kids! Avoid using maya wildcards!

search = nurbsSphereShape*
pattern = |set|*|*|nurbsSphereShape*

cmds = 0.602999925613 sec, 8016 shapes modified


----

import maya.cmds as cmds
import time
import re

pattern = "|set|*|*|nurbsSphereShape*"

search = pattern.split('|')[-1]

# change the maya pattern to a re friendly pattern; | becomes [|], and * becomes .*
repattern = re.sub('[|]',r'[|]',pattern)
repattern = re.sub('\*',r'.*',repattern)

print ("search = %s" % search )
print ("pattern = %s" % pattern )

start = time.time()
sel = cmds.ls( search , long=True)
count = 0
for each in sel:
    if ( re.search(repattern, each)):

       cmds.setAttr("{0}.castsShadows".format(each), 1)

       count = count + 1

end = time.time()-start
print ('cmds = %s sec, %s shapes modified' % (end, count))