Alembic files update pipeline

[Diego Garces]

Hello everybody,

I was wondering how different studios manage their alembic imports, especially regarding the propagation of attributes from the alembic node to the maya nodes (shapes, cameras, etc.).

Are you importing the alembic files in the scenes and reimport them when some attribute change?
Are you referencing the alembic files and any change is always propagated?
Do you have you own references that import the alembic files every-time the scene opens?

The reason I am asking this is because we have built a pipeline that relies on alembic files being updated regularly, and we want this process to be as transparent as possible.

For example, in a lighting scene I have an alembic file of a prop imported and I start lighting the scene. Later on, I discover that one of the shapes have an incorrect subdivision parameter, so I have to reexport the alembic file with the correct value. Ideally I would just need to reexport the alembic file and when I reopen the lighting scene the shape will have the right amount of subdivision.

However, that won't work with the way attributes are handled by the default implementation of AbcImport. I have customized the plugin a bit to connect properties that are not keyframed, so that they can be propagated, but before committing to a workflow very different from the official alembic philosophy I wanted to check how other people solves this problem.

Any suggestion would be greatly appreciated,

Thanks a lot,

Diego

[Tom Price]

Hi Diego,

We handle the Alembic purely through reference, the lighters scenes contain references of the cameras, characters and props. Each of these references contain the alembic Node created by AbcImport and geo / nodes involved. If this is updated, I simply totally recreate the file from the Alembic. We don t version the animation cache as this would be far too many files. So once animation is re-exported that file is made from scratch. (we do this through mayapy.exe).

This means if render Scenes go bad we still have all the assets as they were.

Regarding attributes, do you mean attributes you bake out? in -attr ? I had nt though of linking the attributes that were nt changed through animation.
I am curious how people handle shading? I record the shaders applied to transforms and apply to the objects after the AbcImport in the created reference files.
I know there is now shader support but for what renderers is this implimented? We are using mayatoMr and I can t see any way of dragging the shaders and their networks.

Hope this answered a question and possibly raised some more ?

Regards,
TomP

--
You received this message because you are subscribed to the Google Groups "alembic-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alembic-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

BAFTA Independent Production Company of the Year - 2012 Children's BAFTAs

twitter.com/blue_zoo | facebook.com/bluezooanimation | vimeo.com/bluezoo

blue-zoo, Chesterfield House, 385 Euston Road, London NW1 3AU
t. 020 7434 4111

[Koen]

I was just looking into something like this, switching over the lighting pipeline to use alembic caches. My preferred solution would be more along the lines of the way houdini or the maya gpu cache deal with this. To have a abcRead node that just points to an abc file. 

Ideally I would like to mimic the gpu cache workflow, just load the geometry on the gpu, then add the ability to assign shaders, and rendertime just export a ribarchive like setup (depending on the renderer). The houdini implementation is very elegant, in that what I described is exactly the default alembic workflow, except that you can convert the gpu cache to "real" geometry at any time if you would need it.

I think the first step would be to create an alembic reader that reads the abc "live", or is that possible already?

Cheers,

Koen

 

[Michel Lerenard]

On 05/21/2013 01:13 AM, Koen wrote:
> I was just looking into something like this, switching over the
> lighting pipeline to use alembic caches. My preferred solution would
> be more along the lines of the way houdini or the maya gpu cache deal
> with this. To have a abcRead node that just points to an abc file.
>
> Ideally I would like to mimic the gpu cache workflow, just load the
> geometry on the gpu, then add the ability to assign shaders, and
> rendertime just export a ribarchive like setup (depending on the
> renderer). The houdini implementation is very elegant, in that what I
> described is exactly the default alembic workflow, except that you can
> convert the gpu cache to "real" geometry at any time if you would need it.
>
> I think the first step would be to create an alembic reader that reads
> the abc "live", or is that possible already?
>
> Cheers,
> Koen

What do you mean by reading the abc "live" ?

[koen]

Like in the maya gpu cache or houdini, the node that brings in the alembic file has a file attribute that is set the to the abc. When the geometry is requested, it reads this data from the alembic file. I have not looked at this extensively, but it thought the alembic importer loads the whole alembic file as maya geometry and then looses the link with the abc. So you get a huge .ma file that needs to dump all the geometry to disk again to pass onto a renderer. The nice thing of keeping the link to the abc file, is that you have small scenes, and versioning is very easy.

Cheers,

koen

[Brad]

We are creating a custom maya node that acts similar to rib archive and uses our custom procedural and material file to bind shaders. 

We're also designing the maya node to read the dag structure of the abc to be viewed in maya weather the geometry is loaded or not. 

Last we are adding ability to view geometry in maya without actually saving geometry in maya file (less bloated maya file as you note) 

This is a pretty obvious path for alembic and I think there would be a huge benefit to have it open source. 

Sounds like a few of us are on the same path.

Sent from my iPhone

Brad W. Falk

Brad....@gmail.com

www.lollipopshaders.com

--

[Michel Lerenard]

Ok, you mean streaming the data and not loading everything in memory.

I don't know what kind of alembic reader you're talking about (standalone, maya plugin ?), but that's possible, that's exactly what's I've done: when I import an alembic file, i create objects referencing a node of the file using its object name. Some attributes are read from the start (pivot, animation type, uv presence, etc...), remaining data is streamed only when needed for display or render, it helps having a low memory footprint on huge projects.

--

[Alex Suter]

We stream the transform and geometry data from the cache on the fly as well. The big thing to avoid is re-traversing the entire hierarchy every time you evaluate. We got a big speed up by parsing the hierarchy once, creating the objects in our software, then keeping a handle to the Alembic IObjects so we can jump right to it.

         -- Alex

—
Sent from Mailbox for iPhone

[Moritz Moeller]

On 22/5/13 9:33 AM, Michel Lerenard wrote:
> Ok, you mean streaming the data and not loading everything in memory.
>
> I don't know what kind of alembic reader you're talking about
> (standalone, maya plugin ?), but that's possible, that's exactly what's

> I've done: when I import an alembic file, [...]

This debate makes me wonder ...

It seems everyone is doing more or less the same, sensible thing (as
always).
So I wonder why there is no OSS project on github or SF with a Maya node
that e.g. loads a hierarchy and other wise just uses IObjects
references. Maybe even a bit of scripting framework to turn this into a
mini pipeline.
I mean why does this industry manage to solve some common problems for
everyone by releasing code (EXR, Alembic, Partio, Cortex, OpenColorIO,
OSL, etc) but other problems, that are just as common (referencing ABCs
+ hierarchy & custom attributes, live, into Maya/<insert other 3D app
here>) are still left to be solved again and again, by each studio.


.mm

[Michel Lerenard]

Alembic's goal is not to be a pipeline, but to efficiently store data.
It's not a bad idea to built a smart OS Maya node, but could it fit
studios needs enough to be widely used ? I guess studios developing
their own nodes have them tailored for their particular need(s).

[Moritz Moeller]

On 23/5/13 2:27 PM, Michel Lerenard wrote:
>> So I wonder why there is no OSS project on github or SF with a Maya
>> node that e.g. loads a hierarchy and other wise just uses IObjects
>> references. Maybe even a bit of scripting framework to turn this into

>> a mini pipeline. [...]

> Alembic's goal is not to be a pipeline, but to efficiently store data.

This is a misunderstanding. I am not talking about Alembic or the
Alembic devs. :)

I am talking about everyone else here, the users of this lib.

Why not start a shared project that solves a problem everyone seems to
be having (when they use Alembic) and solve it once and for all.
At least making a very good template node that works out-of-the box and
can then be customized to each studio's needs, in-house.

Ofc every place will have different requirements. But the granularity at
which these differences exist is usually much finer that the problem
that this thread is about. :]

This was a request to start an effort to code share.

That's e.g. how Cortex came into existence. One guy (Andrew Chapman)
asked everyone and someone listened (Image Engine) and made their stuff OSS.


.mm

[Brad Falk]

Exactly! I am in tune with writing something that handles this. Even if its outside of Toonbox actually.

Brad

.mm

--
You received this message because you are subscribed to the Google Groups "alembic-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alembic-discussion+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Brad W. Falk

415-328-5727

bfa...@lollipopshaders.com

www.lollipopshaders.com

Follow us on: 

  

[Francois Chardavoine]

I think there's maybe some confusion about what's available in maya, what isn't, and what's eventually possible.

Maya doesn't allow* a node to create data outside of itself: it can only read data from its input plugs and set data on its outputs (and optionally draw itself). This means an "alembic node" can't create new shapes, new attributes or new connections: you need a separate command for that which will create and connect what's needed. That's what the AbcImport command is, in the two modes it provides ("connect" to existing shapes and attributes or completely "import" what's in the alembic file by creating new shapes).

*when I say doesn't allow: it's technically possible, it's just not "legal" from a maya node framework point of view (and you're asking for trouble down the road by doing that).

So the 4 following options are available, all of them "streaming live data" from an alembic file (and the animation data itself never gets saved in the maya file):

• AbcImport default: this creates the shapes, xforms, attrs from the abc file, and connects an alembic node to them to live-stream any animated data from disk. Since this creates entirely new shapes, things like shader assignments, custom attrs, etc. won't apply.
• AbcImport -connect: this connects to existing scene xforms and shapes, and is a mechanism to update a maya scene to point to new animation abc data, for instance. Custom assignments and attrs on existing nodes are preserved. This also ends up streaming animated data "live" to the shapes and attrs
• Abc reference import: with Alembic 1.1.3 we added FileTranslator support, which allows you to open an abc file and bring it in as a reference.This behaves as you would expect, still streaming live animated data off disk, but has the added benefit of the final shapes not being saved in the final maya file, and any edits/connections/assignments you've made to the xforms and shapes are preserved when you swap the reference to a new alembic file.
• gpuCache: simply draws the contents of the alembic scene in OpenGL, which no way to interact with it. Again, data is streamed live off disk, and the shapes/xforms aren't saved in the maya file.
  

It seems most of what's being brought up is covered by the FileTranslator option which loads an alembic file as a maya reference. Other than custom loading options (like selectively loading the transform hierarchy only without any shapes, etc.), what isn't covered by one of these cases?
Note that I'm only addressing the maya-specific points that were brought up - each application has distinct frameworks or limitations which affect what's possible.

Francois.

[Jonathan Gibbs]

Well, Alembic is pretty young itself, so other projects built on top of it which are that specific probably just haven't happened yet. Even if a studio has a project like this they are considering opening, it's probably still pretty young even in-house. 

Image and geometry formats, partio io, color handling, shading, were also all solved in house again and again for a long time until someone had a system they both thought worth sharing (general and robust enough) and were allowed to share. Neither are trivial!

We opened VDB - it's our 6th volume format since I started here(*). We think we finally nailed it, so it was the right time to go open.

--jono

(*) that's about 0.4 new volume formats per year.

--
You received this message because you are subscribed to the Google Groups "alembic-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alembic-discussion+unsub...@googlegroups.com.