Copy-on-open behavior

[Michael Cleaver]

Hi all,

I was wondering what the motivation was for having the "copy before opening with delayed load" behavior enabled by default.  Is it just because the expected usage is so commonly to open, edit, then re-save a VDB file?  Or are there platform issues (e.g. reasons why having multiple memory maps to the same file is bad?) or other issues? 

In my code I've used setCopyMaxBytes(0) extensively as my usage is that VDB files are never updated/rewritten.  This change however causes openvdb to make a large number of calls to Archive::getLastWriteTime() which, at least under windows, is actually slower than copying the files.  These calls are made when reading an any buffer to check if the file has been modified on disk since the grid was opened.  Again in my case I can assume that they haven't so I've resorted to a hack currently to disable the last write time check in windows.  

I really just want to understand if I'm going to cause trouble for myself by disabling this copy behavior.  Has anyone else done this?

Side note:

I'm planning to test some different implementations of last write time (sadly though std::filesystem is not available for openvdb since it aims to support c++14) and will submit a pull request if I find anything that works better than the existing code in the screenshot.  For my usage commenting this out made my process execution several times faster although most people probably aren't opening and closing as many VDB files as I am so this performance issue may only apply to me.

Thanks,

Mike

[Dan Bailey]

Hi Mike,

Apologies for the slow response to your question. We've talked about removing this copying feature within the TSC actually. The threshold for when files are copied or not is rather arbitrary. I believe that the original intention of adding this feature was not platform-related and was simply to avoid the issues associated with attempting to load buffers from a file that has since been changed.

Your hack sounds reasonable to me. Most of OpenVDB has been developed with Linux as it's primary platform, so it wouldn't surprise me if there are platform-specific issues related to this functionality on Windows that could be improved.

To be more specific about what we'd like to change - we've talked about removing the fine-grained ability to lazily load each leaf buffer. My understanding is that this would mean (1) loading just the header, (2) loading the header and the topology, (3) loading the header, topology and value buffers. For instance, for (2) this would mean memory-mapping the value buffers that could be loaded on-demand in a single operation rather than as needed by each leaf node. To accelerate this, we would then load this data in the background in small chunks into a temporary buffer and thrown away so that it would be then cached by the filesystem. This is the essence of the idea, but we haven't gone into detail or assessed whether this is a workable solution for different platforms yet.

Regarding the use of std::filesystem, we will be releasing OpenVDB 8.0 sometime in the next few months which will adhere to the VFX Reference Platform that allows us to use C++17. Although we still need to be compatible with C++14 for the next couple of years, for a fairly self-contained feature like this, I wouldn't have a problem with us starting to develop an improved version that used C++17 gated by a define if you'd be interested in contributing to something like this. There is still of course the challenge of working with memory mapping in a platform-agnostic way in C++, but ideally we'd also like to remove the boost::iostreams dependency.

Cheers,

Dan

[Michael Cleaver]

Hi Dan,

Thanks for the detailed reply.  In the end I did push ahead with the hack and we've been running it in our production environment for a few weeks without any major issue.  We did end up finding a few spots in our code where we had to call 'readNonresidentBuffers()' to detach from the source file where we wanted to replace it but those were pretty quick to find.  

I'd love to be able to contribute something back to openvdb so I'll try to develop a new C++14 friendly last modified time check for windows.  I'll have a look at the gated option idea too.

RE: upcoming change idea - I still need to get my head around this properly.  Any changes in IO design are of great interest to me since our system has a repository of terrabytes of VDB files!

Thanks,

Mike