Setting and Manipulating Keys Very slow in Referenced Model

[Enrique Caballero]

Hey everyone,

  I am running into a distressing problem.  

An animator just came up to me and complained about it being very slow to key their animation.

When they select all of the controls on the rig and drag the keys around or simply set a key there is a fairly major delay.

I just did some testing and verified that it is quite slow. 

I made the referenced model local, and the speed of setting or manipulating keys increased quite dramatically.

We are using GEAR rigs on this project, which I've never had an issue with before, and I don't really think its gears fault. Maybe its something I layered ontop.

Has anyone run into this problem before?  Our pipeline is heavily dependent on referenced models and I have no plans to use local models in our pipeline at all.  I'm hoping that its just one little oversight that is causing this.

Any insight would be appreciated.

-Enrique

[Enrique Caballero]

Sorry this is in softimage 2013 sp1

[ivan tay]

Hi Enrique,

Do you have a scene file for this ?

Thanks
Ivan

Email : ivan...@nospam.autodesk.com  (please remove nospam from email)

[Enrique Caballero]

Hey Ivan, 

  Thank you, Yep I do, I will send it to you in a few minutes, just packaging up the referenced models

[Ivan]

Thanks!

Sent from my iPhone, please excuse for typos.

[Raffaele Fragapane]

How many custom attributes and specifically some feeding into ICE do you have?

And how many FCurves at a time are we talking about?

We encountered several related issues (and occasionally solved or had confirmation of them, and some QFEs that helped a lot)

--
Our users will know fear and cower before our software! Ship it! Ship it and let them flee like the dogs they are!

[Jeremie Passerin]

I heard the same thing here, and also heard it's much faster in 2014. Have you tested that ? 

[Matt Lind]

I reported an issue in 2013 SP1 where FCurve.SetKeys() was running incredibly slow (like 2500% slower) in the Scripting API.  If the command called by your animator eventually runs a route through FCurve.SetKeys() (your code or Softimage’s), then that would explain the issue.  This was fixed in 2014.

 

To get us working on 2013 SP1, I wrote a wrapper command to expose the same functionality through the C++ API which was not affected by the issue.

 

 

Matt

[Raffaele Fragapane]

There are two issues, a regression, which Matt does a good job of pointing out and that should be fixed in 2014 (to my knowledge, but haven't tested), and other things we found out when a mix of ICE and custom parameters are involved (which is not related to ICE slow at setting them, which was addressed in 2013 already, I believe).

[Enrique Caballero]

well its a gear rig so there are a fair but of custom parameters. but not an obscene amount. And as far as I know, no parameters driven by ICE. although I am using the dual quaternion skinning compound for the envelope.  I've already tested with this removed and it wasnt the issue.

I have tested in 2014 and it is better, by like 30 % but its still massively slower than it would be if the model was in Local Mode.

Seeing as how well Softimage's referenced model and Deltas work together, it is a major reason we use Softimage here at this studio, I would expect the speed of keyframing to be very fast on referenced models, it should be just as fast I dare say, otherwise its a massive hit on their usability.

I have also tested with other rigs as well,  Gear rigs, the malcolm rig, and rigs of my own making, its a pretty major speed difference when setting and manipulating keys on referenced models vs local models. 

As far as how many fcurves.  well... A lot, its an entire rig,  its like 150 objects, and we are keying their entire srt. as the animators are currently blocking animation, so they key the entire character at once. 

2014 is a bit faster, but not massively so. And we don't have the luxury of moving our pipeline to 2014 right now, our tools are written for 2013 at the moment and we are going through a major tools development phase

[Matt Lind]

I would advise training your animators to not key every parameter under the sun.  We had the same problem and had to start slapping wrists as well as be more aggressive with rigging to ensure only parameters that needed to be keyed could be keyed.  That has greatly reduced problems.  It’s not often an animator keys the scaling parameters, so take a close look at the FCurves to see how many of the scaling parameters are actually keyed at values other than 1.0.

 

Matt

 

 

 

 

 

From: softimag...@listproc.autodesk.com [mailto:softimag...@listproc.autodesk.com] On Behalf Of Enrique Caballero
Sent: Monday, May 20, 2013 8:16 PM
To: soft...@listproc.autodesk.com
Subject: Re: Setting and Manipulating Keys Very slow in Referenced Model

 

well its a gear rig so there are a fair but of custom parameters. but not an obscene amount. And as far as I know, no parameters driven by ICE. although I am using the dual quaternion skinning compound for the envelope.  I've already tested with this removed and it wasnt the issue.

[Raffaele Fragapane]

Have you tried changing how the keys are set?

150 objects with the entire local transform set isn't that many curves, we have had issues but that's with thousands piled up on more thousands.

Lastly, is that with the FCurve editor open or not?

I suggest you send the scene to Soft if it can be packaged up.

[Enrique Caballero]

thanks guys,

  yep I've already sent the scene to Softimage. Its definitely a Softimage 2013 Sp1 issue though and not scene related.  

The parameters that the animators can key is already fairly limited as I'm pretty careful with keyable parameters. but I will strip down what i can for now.

It is slow whether or not the Fcurve editor is open.  Basically setting a key on a gear rig referenced model with 180 objects takes 6 seconds, a local model is instant.  

I've stripped down our workgroup to nothing but gear, same problem. 

Really quite distressing! The animators are giving me dirty looks!

[Enrique Caballero]

the keys are being set just by pressing the K key and the keying mode set to "Key all Keyable"

this is the command that gets spit out

Application.SaveKeyOnKeyable()

[Raffaele Fragapane]

Sounds like a regression.

I've had rigs with controls in the hundreds of objects with quite a few added properties and custom parameters, adding up to packs of thousands of keyframes at a pop. It's never been blazing fast, even with rig-centric dedicated commands, but I would have been skinned alive if it took more than a second or two to key out a couple thousand curves.

Consider trying to write your way around it to see how pervasive it is, and if not too far reaching possibly stop them from staring daggers at you.

Keying a ref model will always be slower, there's no two ways about it when you have to deal with delta tracking rather than simply adding data somewhere, but shouldn't be in that measure, it should be just a few percentage points at the most.

[Enrique Caballero]

Thanks raf and everyone for the advice, its really helpful.

I will now reduce their ability to key scaling on the majority of the rig. I will also try to code my way around it, problem is I don't know c++ so I'm stuck with python which I doubt will be able to save me when it comes to slow keying.  

I did test in 2014 btw, and it is a bit faster, but its still massively slow.  

I've sent the scene to Autodesk, we still have a year on Subscription so hopefully they can send us a QFE

[Raffaele Fragapane]

Even with no C++ knowledge you should be able to take different routes around it, just to see if the bottleneck is specifically in one of the wrappers or far enough upstream. Give it a shot.

[ivan tay]

HI,

Had an exchange with Enrique. This is a known issue - logged as SOFT-7029 - Key Referenced Parameter Slower. 

Thanks for bringing this up.

-Ivan

[Jeremie Passerin]

Good to hear that the issue has been reported. This is a big deal for us here too. 

We love Softimage referencing system a lot but are in a situation now where animator are creating local copy of the rigs just to work around the issue, which is obviously a big problem for us. 

The new team has impressed me with all the bug fix they did on 2014. Seeing very old bug fixed is giving me faith in the new team. Keep up the good work, and I'm crossing my fingers to see that one updated in the next release ;-)

[Enrique Caballero]

Yep they are also being incredibly helpful with me. Thanks Softimage Team.  I'm getting our IT guy to fill out the paperwork for a QFE now

[Raffaele Fragapane]

Regardelss of whether one might or might not be happy with the feature list, I think it should be given to the team that they have put some serious effort into keeping in touch with the community and going out of their way to integrate into it, even in the face of some serious negativity (the hostility here and in other places was pretty high in the beginning).

Given that's always been as much a defining trait of Softimage as the software itself might have been, at least a tip of the hat is probably long overdue :)

[Enrique Caballero]

Btw, until we get the QFE, I did a very aggressive removal of keyable parameters and that helped significantly.

Which is something I should have done a while ago.  I was letting them scale every controller in the rig, as I don't like limiting the animators freedom.

Instead i went to the animators, asked them which controllers they usually scale, and then removed the ability from the rest.

This helped me get some speed back.  Its still slow, but much more useable.

Thanks Matt, it was your suggestions

[Enrique Caballero]

Jeremie, I considered letting them animate in local mode but i decided that the risks outweighed the benefits.  

I am just 100% uncomfortable with trusting the animators with Local models. They will start deleting objects and changing heirarchy.

So instead I stripped down the rig of unecessary stuff.  I took all the cloth controls away and seperated it into a different rig, and then aggressively cut down the amount of keyable params.

Its at least manageable now.

Are you really comfortable letting the animators animate local models?  I don't think I could ever let it happen

[Sandy Sutherland]

I did - and we locked everything that was not supposed to move/key.
Worked fine and it is not so difficult to write the tools to do a model
update. Also one BIG plus to this method against referencing is that
you avoid surprises when something changes in the rig that affects
animation and it gets auto updated in a reference, THAT has happened to
me before, largely due to the nature of the overlapping schedules we
work with in SA.

S.

[Raffaele Fragapane]

That's only an issue with live referencing, which is a pretty bad way to go about it.

Versioned referencing will not hold any surprises as you control when a rig reaches what shots. It's a matter of asset management, not of referencing VS localized.

If anything localizing takes a liberty away from you, it doesn't ADD security :)

[Sandy Sutherland]

Yep you are right Raff - but at that time we did not have any asset system going, ask Simon - he and I had the brunt of that.

S.

On 22/05/2013 09:51, Raffaele Fragapane wrote:

[Enrique Caballero]

We do live referencing here as we don't have much for versioning control at this small studio.  Its worked fine for us so far as our rigs are pretty simple and eventually development stops for them on a project. 

Eventually we might do versioned referencing but it would require some asset tracking tools that we just don't have. We also don't have enough coordinators to keep track of what shot gets what rig, or the development man power to update our pipeline.

Tiny shop in Singapore, very difficult to find talented developers.

Until then I've written a bunch of delta cleaning scripts that can solve 90% of our issues, and what was most important for us, was that we are very careful about how we import referenced models so that no unnecessary information gets written to the delta.  So if a rig change does happen, the delta wont freak out and break the rig.

Basically via scripting,  I create an empty referenced model, populate the paths for the rig resolutions, add an empty delta, set the proper settings, disabling the evil ones, and then finally setting the active resolution.  I have found this technique to be absolutely essential to making sure our deltas don't store any extra nonsense in them that would eventually clash with a rig change.  

Otherwise if the animator simply did a vanilla "import referenced model"  this referenced model would have a delta on it thats fully enabled, and it would immediately store a bunch of useless crap under stored expressions/stored positions that would eventually cause the rig to explode when we did the next rig update.

Aside from that, when something bad happens, I run one of 3 delta cleaners if necessary, but to be honest it hasnt happened much in a very long time.

I am very open minded, but i also know the animators at this company very well. If I gave them a local model pipeline they would bring this studio to its knees and I would be out of a job very quickly

[Michal Doniec]

"Eventually we might do versioned referencing but it would require some asset tracking tools that we just don't have."

I'd start with some off the shelf version control software to do half of the work for you (perforce, maybe svn etc.), a rather simple database on top of this and some UI usually does the job for most cases where assets are not too complex and relatively lightweight, ie. mesh/rig character type assets. What I am saying is it doesn't really take much to get a system like this going and benefits can be potentially really big.

Sorry for being bit off topic.

--
----------
Michal
http://uk.linkedin.com/in/mdoniec

[Enrique Caballero]

Hey Michal,

  Your definitely right.  Its something we should look into

-E

[Jeremie Passerin]

I totally agree with Raff. Versionned reference is the way to go. We don't have it here either and it's already an issue on some projects.

I am not happy with animators creating local copy of the rigs, but I'm not with them all day long to check how they work. I discovered they were doing that, I told them it's wrong... 

Cleaning controllers and keyable parameters is my solution too for now... 

[Tim Crowson]

Just to make sure I understand the terminology... when you say 'versionned' referencing, do you mean a workflow that uses controlled 'resolutions'?

-Tim C.

--

 

[Sandy Sutherland]

Tim - it would be a system that controls VERSIONS of rig models for e.g., they would be tested, then passed on to become the 'current' version, which would then update the references.  Basically to avoid say a rigger just writing out the model that is used by a bunch of animators, possibly adding some 'feature' or changing a hierarchy or whatever that then breaks the animation in the scene on the reference model.

S.

On 22/05/2013 17:40, Tim Crowson wrote:

[Eric Thivierge]

Thus you make it so they have no choice but to use the approved method
by only allowing motion to be checked in from ref models or only
allowing importing of the models via a scene assembler UI. :D


Eric Thivierge
===============
Character TD / RnD
Hybride Technologies

[Tim Crowson]

Gotcha. Just making sure the vocab was clear. Yeah that's just asset management then... I have to say we don't have a good system in place for asset versioning either (also a small shop with real constraints), but it's something we're aware that we need.

-Tim

--

 

Tim Crowson
Lead CG Artist

Magnetic Dreams, Inc.
2525 Lebanon Pike, Building C. Nashville, TN 37214
Ph  615.885.6801 | Fax  615.889.4768 | www.magneticdreams.com
tim.c...@magneticdreams.com

Confidentiality Notice: This email, including attachments, is confidential and should not be used by anyone who is not the original intended recipient(s). If you have received this e-mail in error please inform the sender and delete it from your mailbox or any other storage mechanism. Magnetic Dreams, Inc cannot accept liability for any statements made which are clearly the sender's own and not expressly made on behalf of Magnetic Dreams, Inc or one of its agents.

 

[Manny Papamanos]

Hi Enrique.

I repro the issue in 2013SP1 QFE1 x64 with your test scene sent to support.
The keying issue is nonexistent in 2014 x64 with your test scene but the f-curve editing issue is still there.
Also, I tested my own gear character and things are fine with gear and ref models.

I recreated a character keyset on your scene where I included all rotation and all translation on all controllers.
Making it heavier than usual and things still seem to go a lot more smoothly when keying or editing.

You may have omitted a very critical step in your pipeline:

As you may know, 'Key all keyable' is the worst option to use because it will key all the parameters sr+t unless you edit them in bunches in the "keyable parameters editor" under (KP/L).
At the end, inside the keying panel (KP/L), if we take the finger bones as an example, only rotation should be present if you use this option>'Key all keyable'.

I personally use 'character key set' method for blocking then go with marked params...
Key sets are good because you can interactively see what you are keying in one list by selecting the (two keys icon on bottom right)>inspect plus when you open the fcurve editor it reflects all keys in the char key set.

If you have omitted this, then you're looking at 3x more data that has to be processed.
Without this due diligence, the entire scene bogs down because you're dealing with too much data and this probably gets compounded with reference models.
The real time playback will also suffer.

Tips:
Only include controllers in the keying list.
There are a big pile of nulls that are being keyed unnecessarily in your scene.
Also, the little man icon objects, don't include that either.

Most likely someone may be using "key all keyable" and this may be bogging things down.

To be on the safe side:
I would go to the model level and lock what you don't want the animators to animate like all the scaling on all controllers and things like translation on controllers meant to be rotated such as finger bones.

Subsequent scenes will profit from this, I don't know about current ones though.

What does everyone else do to keep things from being keyed in ref model situations in a production environment?


Manny Papamanos
SI and Mobu support

From: softimag...@listproc.autodesk.com [mailto:softimag...@listproc.autodesk.com] On Behalf Of Tim Crowson
Sent: Wednesday, May 22, 2013 1:07 PM
To: soft...@listproc.autodesk.com
Subject: Re: Setting and Manipulating Keys Very slow in Referenced Model

Gotcha. Just making sure the vocab was clear. Yeah that's just asset management then... I have to say we don't have a good system in place for asset versioning either (also a small shop with real constraints), but it's something we're aware that we need.

-Tim

On 5/22/2013 11:55 AM, Sandy Sutherland wrote:
Tim - it would be a system that controls VERSIONS of rig models for e.g., they would be tested, then passed on to become the 'current' version, which would then update the references. Basically to avoid say a rigger just writing out the model that is used by a bunch of animators, possibly adding some 'feature' or changing a hierarchy or whatever that then breaks the animation in the scene on the reference model.

S.

On 22/05/2013 17:40, Tim Crowson wrote:
Just to make sure I understand the terminology... when you say 'versionned' referencing, do you mean a workflow that uses controlled 'resolutions'?

-Tim C.



--



Tim Crowson
Lead CG Artist

Magnetic Dreams, Inc.
2525 Lebanon Pike, Building C. Nashville, TN 37214

Ph 615.885.6801 | Fax 615.889.4768 | www.magneticdreams.com<http://www.magneticdreams.com>
tim.c...@magneticdreams.com<mailto:tim.c...@magneticdreams.com>

[Matt Lind]

>What does everyone else do to keep things from being keyed in ref model situations in a production environment?

 

We used to use parameter locking until we discovered that it doesn’t always work for parameters in referenced models.  A number of built-in Softimage tools don’t function properly with locked parameters either causing tools to abort prematurely (due to lack of error handling) leaving our work in a funky state.

 

We don’t use character key sets because that would force us to create a character key set for every asset in the game the animators touch.  If the rigger forgets to insert a parameter into the keyset, then the animator can’t work until the rigger can update the key set.  Since other artists touch the rigs for different reasons, such as FX artists inserting their effects into the rig hierarchy, that’s a lot of maintenance.  Our custom simulation tools have over 400 parameters per emitter, and sometimes an individual rig can contain up to 15 emitters (rare, but it happens).  That’s too much to maintain without being bitten by human error.

 

Currently we remove the Keyable and KeyableNonVisible parameter flags for parameters that shouldn’t be keyed.  This too is a bit high maintenance, but it’s more acceptable if something goes wrong.  For example, in the case of a character keyset, only parameters in the keyset can be keyed.  If the rigger forgets to add a parameter to the set, the parameter cannot be keyed.  If using ‘keyable’ flag on parameters, most parameters default to keyable being active.  Therefore, if the rigger forgets to remove the keyable flag the parameter is still keyable allowing animators to work.  It’s just a matter of when they key something that wasn’t intended to be keyed that we can go in and quickly isolate the problem and fix it.

 

In both cases there is human error involved leading to undesired results, but using keyable parameter flags results in less time spent on troubleshooting when things go wrong.

 

 

Matt

Magnetic Dreams, Inc.

tim.c...@magneticdreams.com

[Eric Thivierge]

Could you elaborate with some examples of tools that fail? I haven't experienced this myself so I'm wondering which ones they are and which ones to look out for.

 
Eric Thivierge
===============
Character TD / RnD
Hybride Technologies
 

On 22/05/2013 5:43 PM, Matt Lind wrote:

[Matt Lind]

I’ll have to wait for somebody to experience it again before I can specify in great detail.  Usually in the area of enveloping and cluster tools.  I complained a bit about in the past, so maybe some things got fixed.  It was a big problem in 7.5.

 

 

Matt

 

 

 

 

 

From: softimag...@listproc.autodesk.com [mailto:softimag...@listproc.autodesk.com] On Behalf Of Eric Thivierge
Sent: Wednesday, May 22, 2013 2:50 PM
To: soft...@listproc.autodesk.com
Subject: Re: Setting and Manipulating Keys Very slow in Referenced Model

 

Could you elaborate with some examples of tools that fail? I haven't experienced this myself so I'm wondering which ones they are and which ones to look out for.

[Alan Fregtman]

I know one case:

If you put an expression on the View Visibility, and use H to hide the object, Render Visibility rightfully toggles off, but if you tap again, it won't come back on.

Remove the expression from ViewVis and it'll hide and unhide correctly again.

[Alan Fregtman]

Forgot to say, it's the same deal if you lock the param value. (Rightclick on the green square, Locks, Param Value)

[Raffaele Fragapane]

We mostly tend to use software, by version, or patched, or by package, that doesn't shit itself over a meager 700 curves :p

We have the same problem to some extent, but working in the thousands, and when it's addressed it's addressed by giving people tools that key things more conservatively. It's not always done though, and occasionally we pay the blood price for it and ask for more fixes.

As for character key sets, the general consensus, and this is across several apps, is that they are the festering pit where good animation goes to die, so we tend to stay away from them. They require a specificity of actions and knowledge, not to mention a maintenance overhead, that means they simply don't get used by any animator.

You assume that animators work the way you do, with a thorough understanding of the software's intricacies, most of the times they don't.

As far as software interaction goes, you have to assume rolling one's face on the keyboard is as far as basic competence is required, and being able to roll it both left to right and right to left is considered extreme software dexterity and makes one an Autodesk Master.

[Enrique Caballero]

Hey Manny,

  Thanks for looking into it.  I"m going to send you a video later today showing you just how slow it is.  Its more than just having too many parameters set to keyable.

Your point about limiting the keyable parameters is 100% true and I should have locked several of them out. 

But with that said, this rig does not have that many parameters or objects in it to account for the 6 second delay that I experience on every system at this company.

With that said. I have found something interesting.

Last night while we chatted, I tried the scene on my laptop at home, and the issue was largely gone.  

This makes no sense to me as I have done heavy testing here with the workgroup disconnected and a vanilla softimage and gear install. And had the 6 second delay.  

I now think it might have to do with our system configurations here, I am looking into it.

Thanks for the help, Please don't close this ticket yet. The problem is larger than just "too many keyable parameters"

-Enrique

[Enrique Caballero]

Hey everyone,

  So we are currently working with Autodesk to try to find a solution to this issue. We have made this issue the #1 priority for one of our TD's.  And for those of you who are also suffering from this problem

ie. Jeremie Passarin, Rafaele Fragapane.

This is what we have found. 

I am pasting our report below. I hope that we can get this fixed.  Hopefully this will help you guys out as well.

V

_____________________________________________________

Alright so here is the testing we have done so far on the "slow keyframing" issue when trying to set keys on multiple controllers. In this case we are talking about a characters 206 controllers with multiple attributes. As it is, trying to do this will take anywhere between 4-6 seconds on all tested computers here at the studio.

Although we haven't solved the issue I can confirm the following can be ruled out:

- scene files located on the network with referenced assets have no impact on the speed. local scenes with local assets on an SSD have been tested.

- Clean Softimage user profiles were tested + freshly created workgroups were tested so we can rule out our custom tools and pipeline scripts from having an effect. Gear was also tested on a workgroup install and under User Add-Ons install. No impact on speed.

- Possible bloatware from HP or "dirty" Windows system with various background apps can be ruled out. We clean installed a standard workstation with a vanilla copy of windows 7 and Softimage 2013 SP1 + python + gear. No impact on speed.

- Process priority, process affinity, CPU energy saving/downclocking, Windows core parking, hyper threading was verified, edited to max out settings, and has had no impact on speed.

- Tested our dual CPU systems with only 1 CPU, and according RAM removed. Also tested if there were any difference between similar Sandy bridge and Ivy bridge CPU systems, no impact on speed.

---------------------------------------------

Keeping in mind the testing that was listed above, I should note that our definition of "slow" keyframing comes from the fact that we have replicated the conditions of the test scene on our personal computers/laptops and have achieved timings pretty much cut down by half. i.e. 2-3 seconds vs 4-6 seconds.

The only thing these personal computers have in common with each other is a higher core speed than our studio workstations. Although I am hesitant to jump to conclusions, everything is pointing to the fact that keyframing in Softimage is purely dependant on the process running on a single thread.

Most of the studio testing was done on a dual CPU workstation with Sandy bridge Xeon 2Ghz with 6 cores each + 16 Gigs RAM.
I did a test at home last night on my home computer which is a single CPU i7 930 Quad core nehalem with 6 Gigs RAM (just FYI nehalem is a older micro architecture than Sandy bridge). As I overclock, my CPU is running at 4Ghz. I was able to keyframe all the controllers between 1 and 2 secs.

Next, I downclocked my CPU to 2Ghz to match the core speed of our workstations. Keyframing timings matched the 4 seconds it took at work. I disabled 3 out of 4 cores on Softimage process and left the frequency at 2Ghz. Keyframe timing remained at 4 secs. A couple things became clear to me at this point. First of all, it won't matter how many cores you throw at it, since CPU usage during the keyframing only utilizes 1 CPU thread, similar to other various Softimage functions like duplicating. Not that I expect this type of process to be multi-threaded, but I'm just pointing out that a 12 core workstation vs a dual core laptop will have no significant impact on the speed of the keyframing instructions.

-----------------------------------

Again all these results are relative for the moment. For all I know, 2 secs to keyframe 206 controllers on a i7 CPU running over 3Ghz should still be considered slow? hard to say really... Still a couple things I'd like to try, like older ver of Python, maybe older version of gear. Wish I could test the scene on older version of Softimage, sadly scenes aren't backward compatible.

anyways, will keep playing around with this for a bit when I can.

cheers,

-John