many transforms slow alembic down? (houdini/soho export)

[Imre Tuske]

Hiya,

Right now I'm working on a soho implementation of an alembic exporter
(actually the first 'bleeding edge' version is kinda finished). This
was a necessary next step from the plain ROP geometry exporter
( https://github.com/johnnyquest/hAbcGeomExport ) as soho handles all
kinds of geometry instancing in an unified way, and right now we need
that.

So, we have a shot which is a classic futuristic-cityscape-with-flying-
cars, and guess what, the "flying cars" are instances (about 10,000 of
them). I tried two ways:

1. Exported as one big 'deforming' mesh, frame-by-frame: export speed
was constant, file got about ~5gigs in size for ~300 frames (took
about half an hour) -- it works, but the file is too big, single mesh,
etc.
2. Once the soho exporter was working, I exported the same thing but
as instances, which resulted in a file of ~300 megs (pretty cool), BUT
export time went up to 2,5hrs!

So when exporting these 10,000 transforms, export time got quite
longer (and since the geometry themselves are non-deforming, I didn't
export it for each frame, so basically the export process was writing
out just an insane amount of transforms).

Also, the individual frame export times varied between 4-5 secs and 20
secs with no correlation to anything I could notice (no geometry
changes, number of cars are the same throughout and sequence, etc). It
might be some kind of cache-related problem, or hdf-specific stuff, I
don't know.

(I narrowed it down right to the transform export call: if I comment
the one line where it happens ["_xform-
>getSchema().set(xform_samp)"] , per-frame export time will be the
same for all frames.)

Shortly, my export times became quite erratic (and much slower) when
it came to writing _lots_ of transforms. Any ideas on how to fix/work
around this would be really appreciated... thanks!

imre

[Peter Bowmar]

I think you might be stuck, SESI has done work in H12 to "fix"
instance write times, but in H11 Soho based instancing (specifically
the generation of the IFDs) is almost unusable because it's so slow :(

Cheers,

Peter B

> --
> You received this message because you are subscribed to the Google
> Groups "alembic-discussion" group.
> To post to this group, send email to alembic-d...@googlegroups.com
> To unsubscribe from this group, send email to
> alembic-discuss...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/alembic-discussion?hl=en
>
> For RSS or Atom feeds related to Alembic, see:
>
> http://groups.google.com/group/alembic-dev/feeds
>
> http://groups.google.com/group/alembic-discussion/feeds

--
Cheers,

Peter B

--No, I am not on Facebook.

[Lucas Miller]

Other people have noticed this as well outside of Houdini as well.

One technique that can help alleviate the problem is to not have all those xform nodes be siblings of each other.

You'll still have problems for 100k animating transforms, even if they are in a regular hierarchy.

Users have also noticed that it's faster to write all of the xform samples per node, but this usually isn't practical.

If you have a LOT of xform nodes AND are writing it over a long enough frame range, you might be able to split the export up into chunks and then use the abcstitcher utility to merge them all together in one new file.

[Imre Tuske]

On Dec 16, 7:31 pm, Peter Bowmar <pbow...@gmail.com> wrote:
> I think you might be stuck, SESI has done work in H12 to "fix"
> instance write times, but in H11 Soho based instancing (specifically
> the generation of the IFDs) is almost unusable because it's so slow :(

Yep, but that's for exporting to mantra, if I recall correctly.

Actually, getting the instance data through soho is pretty fast, takes
about ~0.2 secs for 10,000+ instances.

It is confirmed by Lucas that alembic can get a bit slow when
exporting this amount of transforms (reading speed is okay). I'll try
some trickery (and/or include a stitching option in the workflow).