Any tips on production use of Alembic?

[Daniel Wong]

Hi guys, 

I and my colleagues have been trying to use Alembic v1.5.2 for Maya on production for a month, but the problems we face make me so frustrated.

1. ABC Cache performs even slower than rigged model

In order to reduce the loading of maya in lighting, we export geometry data as .abc from animated scene. The performance of the cache for models with simple hierarchy is satisfactory. But for models with complicated transformations (e.g. robots with lots of mechanical parts), Maya needs quite a long time to open the scene, (a model of 1 million faces need 1.5 minutes, and we have at least 4 to 5 robots in the scene). Every time Maya crashes, which may not be caused by Alembic, we are punished by spending 6-10 minutes to open the scene again. 

And I'm not sure if Maya reads the data of all of the frames when open the scene, but it still takes long to switch to another frame in timeline.

The performance is even worse than referencing the animated rigged model.

2. Maya does not use full CPU power to process ABC

We are using ogawa as our export format. We have checked the Task Manager when importing ABC files to Maya scene, we find that the CPU usage only rises about 5-10%, and from the chart I even cannot sure whether it is processing with multi-thread or not. The performance is not as what is mentioned in Alembic website (25x of HDF??)

Is there anyone using ABC on production satisfactory? Any tips for us to use ABC? And any potential cause can you think out which leads to the problems we face? 

[Lucas Miller]

What version of Maya are you using?

Do you have an example model you can share, or a rough idea of how man transforms in your scene (and how many of those are transforming)?  AbcImport does NOT read all the frames when opening the scene.

The 25x quote you saw refers to read performance with apps that support multi-threaded reads.  Unfortunately Maya isn't yet one of those apps.

If the CPU only goes to 5-10% while exporting, then it sounds like your rigs for these assets are cheap to evaluate in comparison to the time it takes to write the calculated data to disk.

Lucas

--
You received this message because you are subscribed to the Google Groups "alembic-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alembic-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Daniel Wong]

Sorry for late reply since I work on some testes about loading ABC.

Just as Lucas' guess, the evaluation of model is cheap actually. 

And we are looking for the bottleneck which seems not related to the performance of ABC.

Thank you Lucas!

Lucas於 2014年3月2日星期日UTC+8上午2時35分40秒寫道：

To unsubscribe from this group and stop receiving emails from it, send an email to alembic-discussion+unsub...@googlegroups.com.

[Gael Honorez]

Are you using the gpuCache or the AbcImport command?

GpuCache is bad in the sense that he caches the whole animation even if you never move in the timeline (can be VERY long for heavy meshes/long animations).

AbcImport is fast by itself, but as it has to reconstruct the maya objects and connect to them, and that the maya API is slow as hell past a certain amount of connections (it's really not related to AbcImport, and that amount is actually very low for production scenes these days), well, you have your culprit...

[Ævar]

  A 10 minute loading time sounds like a potential network issue, probably best to eliminate the hardware first:

[01]

  Run the same scenario writing to a local hard drive and your server, if it still takes 7-10 minutes then it's a read issue.

[02]

  Run disk I/O analysis to see if your hard drive is hitting it's read/write limits since the computation itself is low.

This will eliminate the ethernet cables or hard disk limitations as factors in your tests, 

[03]

  If you are then no closer try running the dg profiler while importing ( found in the Maya plugins ) and playing the timeline, this will identify if there are any specific DAG nodees slowing the whole thing down.

[04]

  If at this point you still don't have an answer, your error checking, the hardware, and network is performing great then you can try the cProfile python module to further pinpoint this,it will give you an overflow of evaluation information you can then parse with the pstats module, run it on a load function and you will clearly where the code is taking up the most time or resources, it won't show you the specific part of the cache that's evaluating slow but what calls are being made and what is calling what.

  But then, if there is still nothing, assuming you are on a Linux box, go to gdb.

Good luck finding why it takes so long to load this particular cache, have you tried simply switching out the workflow from an Alembic Cache to an Alembic GPU cache?  Those perform a lot faster but don't meet everyones specs so it may not be the solution for you, in short, a Maya scene should never go as far as take 10 minutes to load, even with a lot more geometry than what you refer to, if it's consistent and the checks above go through fine then you may be in a scenario where your workload has become so much that you need to put the workflow under the microscope, as in use build scripts and proxys rather than artists dry loading scenes.

  Hope this helps

[Daniel Wong]

@Gael: Thanks Gael. I'm using Maya 2014, and I just "create reference" to reference those .abc, just as referencing .ma/.mb

@AEvar: Thanks! You've given me so much idea to debug it! I've tried loading the scene locally and it's a lot faster. And it needs only ~15s to load it. It's really the issue of network.

I've checked the network traffic it doesn't touch the limit at all, it seems that it maybe due to the slow response of our server. I am not sure if it is caused by users "working directly" with servers.

I know that it inflicts more server I/O when using softwares to read from / write to server directly, however it is very inconvenient if all the data should be uploaded / downloaded before and after processing. (much more scripting will be needed to handle those things, and also synchronization, reference path switching...) On the other hand, it is not a good idea for users to take care of those subtle server paths and local paths.

May I know if you always work with ABC (or any other read/write action) on server directly?

Daniel Wong於 2014年3月1日星期六UTC+8下午1時25分20秒寫道：

[Ævar]

Sorry for the late reply

  Most studios write their cache's straight to a server as far as I'm aware of but it's terribly bad sport.  Can provide at least 15 arguments for why that is a silly idea but this one is my most favorite:

  7. The difference in transfer capabilities between an ethernet cable vs the read/write performance of your hard drive, let along if you have an SSD is not even worth discussing.  Proposed workflow for anyone writing directly to a server would be something along these lines:

[Before]

01 - Press "cache"

02 - wait while GUI hangs and hope nothing breaks as the file is getting generated over a network.

03 - If all went well then return to work afterwards and try to ignore the hours of accumulated time lost on daily basis when there are 20-50 people pressing that button.

[After]

01 - Press "cache"

02 - wait a few seconds while the file is written out at the speed your hard drive handles.

03 - Return to work

04 - a background process runs which syncs your local exports to the server.

  The second examples puts the delay on the receiving end where it doesn't matter if the file arrives this minute or a little later, the workstation which the artist is working on stays free during and no time is lost on waiting.

  Hope this helps

[Michael Jefferies]

We use Alembic in our pipeline for the past two years, but I wrote a custom maya plugin that does the alembic importer/exporter and gpu cache reader.  This was all before ogawa, so some things we had to do to improve performance were the following.

* On export we write out the regular cache and a "merged" cache which has all maya meshes merged into a single alembic mesh node.
* For import there are two modes - a regular mode that imports caches which drive maya meshes, and a gpu cache mode which is a custom locator that reads the caches and displays them directly using OpenGL.
* If you combine the merged mesh cache export with the gpu cache custom locator you get the best performance, although it is useful only for interactive and playblast use.
* We have scripts that swap the regular cache with the gpu cache and vice versa, so that you can swap in order to render, since it needs the regular cache that drives the maya meshes.

[Derek]

On Friday, February 28, 2014 9:25:20 PM UTC-8, Daniel Wong wrote:

The performance of the cache for models with simple hierarchy is satisfactory. But for models with complicated transformations (e.g. robots with lots of mechanical parts), Maya needs quite a long time to open the scene,

Specifically for the example of a robot, if the geo is moving with transforms rather that deforms you can do a little trick on the rig that will make it a lot faster:

 Instead of parent constraining the geo to your rig, instead make a group above it and constrain that. Then before you export to Alembic, hide the geo and export with the "renderable only" option checked. This will export just the groups with your animation, but not the hidden geo. That makes the file size is a lot smaller. You then import-merge that cache into your geo file. Should be very fast.