Yup it really is because of the lambda function... I change the apply button command to this:
applyButton = pm.button( height=buttonHeight, label='Apply', command='avRandomizer._windowApply("%s", %d)' %(window, False) )
baking the command into a dumb string, and the undo work as expected.
I am suspecting that the lambda function is going through the code, building some sort of huge command list. When I call the lambda function Maya is not recording one command but many commands, thus giving me this undo problem.
I would still really like to be able to use lambda function in my window command, because sometimes sending a command as a string is not possible... like if I want to send a list to my function, building a string might prove problematic.
--
They say, "Evil prevails when good men fail to act." What they ought to say is, "Evil prevails."
Nicolas Cage as Yuri Orlov in Lord of War.