Memory optimisation assisted by Alembic files?

[Nicholas Yue]

Hi,

Modern renderers are generally path tracing and global illumination class. The traditional method to manage peak memory for geometries like points and curves via bounds and delay archive are no longer effective as renderer tends to load all the geometries.

I am resurrecting some old ideas I pondered but never built, I have a couple of questions:

1. Can an Alembic file be open as a mmap file, allow us to pull in only what is needed and allow the OS to drop stuff when memory is tied?
2. Does the layout of the geometry within an Alembic file i.e. index ordering, space filling curve, helps with caching and memory coherence
3. Can a re-indexing also benefit from LOD, maybe even Continuous LOD?

Kind regards.

--

Nicholas Yue
https://www.linkedin.com/in/nicholasyue/

[Jim Price]

Modern Alembic is read by mmap and loads only the file pages the renderer requests. Assuming the renderer doesn't overrequest samples, the I/O overhead is quite efficient. And yes, the layout of your geometry in your Alembic can have a significant impact on rendering I/O.

The talk covers how a production went sideways because of Alembic's mmap and how the files were written out. If I recall correctly, Alembic doesn't specify an madvise, so you get MADV_NORMAL by default which will load 32 pages worth of data (128 KB) around the requested data read. This is normally fine if all your sample data for a frame was written out together, as is usually the case in Houdini where it saves frame by frame. But when using tools like the abcstitcher which writes out data object by object, instead of frame by frame, you can run into cases where to read one frame's worth of data with the default mmap MADV_NORMAL will cause your entire file to be read in. In one production, the required data for a frame should have been around 700MB, but the I/O overhead was about 11GB due to the overreading caused by mmap. Our fix for this was to add an env var, that would set an madvise of MADV_RANDOM, which only reads the specific pages requested.

--
You received this message because you are subscribed to the Google Groups "alembic-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alembic-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/alembic-discussion/CADYXiJ%2Ba_vNj6dNvoGJ%3DOH7ZMvkk0oYPnJTpbQqfYn3taLqg7A%40mail.gmail.com.

[Lucas Miller]

For cases like abcstitcher, why not read those with pread? (via kFileStreams in IFactory)

How careful are you about making sure your render only reads properties that will actually be used at render time?

For example it can be very easy to write out several per point attributes from the DCC that you may not actually need at render time.

Lucas

To view this discussion visit https://groups.google.com/d/msgid/alembic-discussion/CALgZDefKq3MgKWKe6_RoZLZ9UJeCg%2B4On0e9ARH3hC-ZiAvMkw%40mail.gmail.com.