single thread lag before multithreading with foreach.

[Greg Hurst]

Let's say I want to modify every value on my grid by multiplying each by 2. I can define a functor and pass this and my tree to foreach(). However I notice for large grids there is a long lag before any multicore processing begins. 

Here's a simple example:

struct ScalarTimes {

    float fac;

    ScalarTimes(const float s): fac(s) {}

    inline void operator()(const openvdb::FloatGrid::ValueAllIter& iter) const {

        iter.setValue(fac * (*iter));

    }

};

openvdb::FloatGrid::Ptr grid = openvdb::tools::createLevelSetSphere<openvdb::FloatGrid>(1.0, openvdb::Vec3f(0.0, 0.0, 0.0), 0.0005, 3.0);

openvdb::tools::foreach(grid->beginValueAll(), ScalarTimes(2.0));

This example creates a level set with ~300,000,000 voxels, and on my machine the call to foreach() takes ~17 seconds. However on my 56 core machine only 1 thread is fired up during most of the computation.

Here is a plot of cpu usage over time (5600% is the max):

Is this expected behavior? And is there a way to start multithreading earlier?

[Nick Avramoussis]

This does seem odd as both methods are multithreaded. Are you able to pinpoint on your graph exactly where tools::createLevelSetSphere and tools::foreach start and end?

[Dan Bailey]

Yes, this is to be expected. What's happening here is that on calling openvdb::tools::foreach(), a tree::IteratorRange is being created:

https://github.com/AcademySoftwareFoundation/openvdb/blob/master/openvdb/openvdb/tools/ValueTransformer.h#L322

This is very slow with a "large" iterator such as a value iterator as it must iterate a long way through the range before it can split to use more than one thread and that's what you're seeing here. The answer is to only use tools::foreach() with smaller iterators such as leaf iterators or otherwise to use a different technique such as a LeafManager to achieve the same thing which doesn't have this performance trade-off. This tools::foreach() method is primarily provided for convenience.

FYI, there is an OpenVDB Siggraph course this year that covers a variety of topics. In my section of this course, I present a variety of different sequential and parallel algorithms in OpenVDB and analyse their performance and when to use each one:

https://s2021.siggraph.org/presentation/?id=gensub_281&sess=sess155

[Greg Hurst]

Nick: I should have mentioned that the plot only profiles the call to foreach().

Thanks Dan! Yes I'm seeing much better performance both with NodeManager and LeafManager. I look forward to this coarse and the nanoVDB talk :)