Undoing scripted plugin calls from MFnMesh?

[AK Eric]

I wrote a tool (Python function, not scripted plugin) that assigns
vert colors via the API, through MFnMesh.setVertexColor(). Works
great, really fast. Problem is, it won't undo.
Unless I'm missing something, I've deduced that when API calls modify
the DG (add\delete nodes\connections, color verts, etc) unless they're
authored via a scripted plugin supporting undo (via MDGModifier), you
can't undo them. If anyone has a workaround for /that/, I'd love to
hear it. But I digress...

So I went about re-authoring this function as a scripted plugin so I
can support undo. However, it's still not undoing... properly. It
"sort of undo's", more on that below.

Other scripted plugins I've written that use the functionality of the
methods on a MDGModifier (create, rename, delete, etc), I can get them
to undo just fine:
http://download.autodesk.com/us/maya/2010help/API/class_m_d_g_modifier.html

But the MDGModifier has nothing for setting vert colors, that's where
MFnMesh comes in obviously. MFnMesh has a setVertexColors() method:
http://download.autodesk.com/us/maya/2010help/API/class_m_fn_mesh.html#7226f44a9d5d14689a8f1f7c822d6d0f
It can optionally take a MDGModifier reference. But if I pass one in
or not seems to have no effect on the undo operation: No undo
whatsoever.

So as a workaround, I've gone about authoring code that will remember
the previous vert colors & store them (during redoIt()), so that when
the command is undone (and the undoIt() method is called), that state
is restored. And it works... sort of. The first time yes, but after
that it gets pretty hairy.

Any thoughts on this? Anyone been successful at passing a MDGModifier
reference to a MFnMesh.serVertexColors() method and getting undo to
properly work?

thanks

[Justin Israel]

While I cant comment in terms of direct experience with this, it seems that according to the docs, passing the modifier SHOULD populate it with the undoable command. 

This is kind of a weird shot in the dark but its work a try. I noticed for some reason the docs omit referencing the modifier input on setVertexColors, whereas on setVertexColor it is included. It makes me wonder if the setVertexColors is possibly ignoring the modified parameter. Have you tried iterating over your vertex list and calling:

MStatus MFnMesh::setVertexColor	(	MColor &	color,
		int	vertexIndex,
		MDGModifier *	modifier = NULL
	)

... just for poops and giggles?

But then again, the approach of maintaining the previous values manually is pretty much the purpose of the undoIt()  in the command. So if this really doesn't work with the modifier.undoIt(), I think its totally appropriate to remember the state and manually restore it, which is what you would have to do in any command if you werent using the MDGModifier.

--
view archives: http://groups.google.com/group/python_inside_maya
change your subscription settings: http://groups.google.com/group/python_inside_maya/subscribe

[AK Eric]

Thanks for the reply: It appears, based on my experimentation, that
the MDGModifier has no real effect on MFnMesh.applyVertexColors():
When the colors are applied, they're applied *right then*, not via the
MDGModifier.doIt() method. I find this questionable, like I'm missing
the sceret handshake or something...

I re-wrote the whole command to store the pre\post states and manually
deal with the undoing, and now it's quite happy (and makes no use of
MDGModifier now). Live and learn.