Problem with ParticleToLevelSet slow down over time

[Anthony Grafft]

I'm working on a particle system that uses OpenVDB, but I've noticed a slowdown specifically when using OpenVDB's ParticlesToLevelSet tool. I've been noticing the more that I use the tool the more that it's speed decreases so I made a test project for it to try and figure out what's going on. 

for(int i = 0; i < 20; i++)
{
       setStartTime(blockTime);
       openvdb::FloatGrid::Ptr cachedLevelSet = (openvdb::createLevelSet<openvdb::FloatGrid>(voxelSize, halfWidth));


       openvdb::tools::ParticlesToLevelSet<openvdb::FloatGrid, openvdb::Index32> raster(*cachedLevelSet);
       raster.setRmin(1e-6);
       raster.setGrainSize(1);
       raster.rasterizeSpheres(myParticles);
       raster.finalize();

       dump("ParticlesToLevelSet index = %d took %.2lf sec\n", i, getEndTime(blockTime));      
}

Basically every time I run that loop I'm seeing the speed increase gradually every iteration and then it flattens off. With the particles that I'm loading for the test I'm seeing the first iteration to take about 5.5 seconds and then every iteration after it builds to 6.5 seconds and then sits pretty consistent +- 0.1 seconds of that. I thought it had something to do with flushing the grid I'm making before hand but even clearing the grid every iteration seems to make no difference with no memory leaks.

I'm just wondering what can be done to prevent this slowdown and keep it more consistent with the original starting speeds for ParticlesToLevelSet ? Or is this slow down to be expected when utilizing  ParticlesToLevelSet ? 

[Ken Museth]

Hey Anthony,

I think I know what's causing the seemingly strange behavior your describing, but I obviously can't be sure without knowing exactly what your benchmark code is doing :) The problem is likely that you're repeatedly rasterizing particles into the same grid. In the first iteration the grid is empty (doh), but during subsequent runs the grid already contains a level set and my algorithm is now designed to take a different code-path in which it will union the contribution of the particles with the existing level set surface. This incurs an additional overhead which is consistent with what you're reporting. Assuming this is indeed what you code does the solution is pretty straightforward, simply make sure every run rasterizes into an empty grid - either by instantiating a new grid per run or by clearing a single grid between runs (but obviously outside the scope of the CpuTimer :)

Hope this helps
Ken

[Anthony Grafft]

Hi Ken,

Thanks for the response! So I've tried using clear in the past after rasterizing in the loop and also with a completely new grid every iteration but I still see that slowdown. As for the rest of the code not being shown for the test is just loading particles into the myParticles class from a file of about 4.2 million particles.

Here is the modification I made to the loop:

for(int i = 0; i < 20; i++)
{
       setStartTime(blockTime);
       openvdb::FloatGrid::Ptr cachedLevelSet = (openvdb::createLevelSet<openvdb::FloatGrid>(voxelSize, halfWidth));


       openvdb::tools::ParticlesToLevelSet<openvdb::FloatGrid, openvdb::Index32> raster(*cachedLevelSet);
       raster.setRmin(1e-6);
       raster.setGrainSize(1);
       raster.rasterizeSpheres(myParticles);
       raster.finalize();

       dump("ParticlesToLevelSet index = %d took %.2lf sec\n", i, getEndTime(blockTime));

       cachedLevelSet->clear(); // << clear added here

       if(cachedLevelSet->empty()) // added to confirm grid is empty when checking cachedLevelSet->empty()
           dump("EMPTIED\n");

}

Here is the console read out so you can see the behavior:

UpdateParticles took 1.87 sec

Particle Count = 4242109

ParticlesToLevelSet index = 0 took 5.51 sec

EMPTIED

ParticlesToLevelSet index = 1 took 5.62 sec

EMPTIED

ParticlesToLevelSet index = 2 took 5.68 sec

EMPTIED

ParticlesToLevelSet index = 3 took 5.70 sec

EMPTIED

ParticlesToLevelSet index = 4 took 5.80 sec

EMPTIED

ParticlesToLevelSet index = 5 took 6.27 sec

EMPTIED

ParticlesToLevelSet index = 6 took 6.21 sec

EMPTIED

ParticlesToLevelSet index = 7 took 6.15 sec

EMPTIED

ParticlesToLevelSet index = 8 took 6.32 sec

EMPTIED

ParticlesToLevelSet index = 9 took 6.27 sec

EMPTIED

ParticlesToLevelSet index = 10 took 6.44 sec

EMPTIED

ParticlesToLevelSet index = 11 took 6.35 sec

EMPTIED

ParticlesToLevelSet index = 12 took 6.51 sec

EMPTIED

ParticlesToLevelSet index = 13 took 6.50 sec

EMPTIED

ParticlesToLevelSet index = 14 took 6.42 sec

EMPTIED

ParticlesToLevelSet index = 15 took 6.44 sec

EMPTIED

ParticlesToLevelSet index = 16 took 6.45 sec

EMPTIED

ParticlesToLevelSet index = 17 took 6.76 sec

EMPTIED

ParticlesToLevelSet index = 18 took 6.93 sec

EMPTIED

ParticlesToLevelSet index = 19 took 6.67 sec

EMPTIED

Press any key to continue . . .

As you can the time is still increasing, but if I drew it out say 50 iterations it levels off. 

Just to be clear, am I misunderstanding what you're referring to when clearing the grid after rasterizing and there is another function besides the one I'm using above? 

Thanks

Anthony

[Ken Museth]

Since I don't know what timer you're using can you try to replace
setStartTime(blockTime);
with
openvdb::util::CpuTimer timer("ParticlesToLevelSet");// remember to include <openvdb/util/CpuTimer.h>
and

dump("ParticlesToLevelSet index = %d took %.2lf sec\n", i, getEndTime(blockTime

));
with
timer.stop();

Also, what is your voxelSize, the size of the input particles in units of the voxelSize, the OS, the core count of your CPU and the compiler version? Finally when you build are you linking against jemalloc or tbbmalloc - or neither? All of these factors could affect your performance. The best would of course be if you could email the actual code (preferably simplified as much as possible) so I try to reproduce your observations. In short I'm baffled :)

Cheers,
-K


[Anthony Grafft]

Went ahead and sent an email, thanks!

[Anthony Grafft]

Hello, I'm still receiving this issue of ParticlesToLevelSet gradually slowing down more every iteration I use it and eventually caps off. I turned it into a pretty bare-bones test. The increase tends to cap at about 20% in time taken for ParticlesToLevelSet, so ideally I'd like to reduce this if possible for larger particle data sets.

Has anyone seen this issue before or know of a possible fix?

[Ken Museth]

Hi Anthony,

I just ran your test (the one you emailed) on a (small 4 core) Ubuntu 16.04 box and I didn't observer the slow-down you reported. I suspect it is related to the fact that you're running your test under Windows (?!). Anyway below are the timings I got:

ParticlesToLevelSet ... completed in 5 seconds and 645.6 milliseconds (5645.6)
ParticlesToLevelSet ... completed in 5 seconds and 139.8 milliseconds (5139.8)
ParticlesToLevelSet ... completed in 5 seconds and 128.2 milliseconds (5128.2)
ParticlesToLevelSet ... completed in 5 seconds and 136.5 milliseconds (5136.5)
ParticlesToLevelSet ... completed in 5 seconds and 192.7 milliseconds (5192.7)
ParticlesToLevelSet ... completed in 5 seconds and 164.8 milliseconds (5164.8)
ParticlesToLevelSet ... completed in 5 seconds and 220.9 milliseconds (5220.9)
ParticlesToLevelSet ... completed in 5 seconds and 233.2 milliseconds (5233.2)
ParticlesToLevelSet ... completed in 5 seconds and 165.7 milliseconds (5165.7)
ParticlesToLevelSet ... completed in 5 seconds and 132.3 milliseconds (5132.3)
ParticlesToLevelSet ... completed in 5 seconds and 235.3 milliseconds (5235.3)
ParticlesToLevelSet ... completed in 5 seconds and 133.3 milliseconds (5133.3)
ParticlesToLevelSet ... completed in 5 seconds and 217.6 milliseconds (5217.6)
ParticlesToLevelSet ... completed in 5 seconds and 149.0 milliseconds (5149.0)
ParticlesToLevelSet ... completed in 5 seconds and 229.7 milliseconds (5229.7)
ParticlesToLevelSet ... completed in 5 seconds and 231.6 milliseconds (5231.6)
ParticlesToLevelSet ... completed in 5 seconds and 149.7 milliseconds (5149.7)
ParticlesToLevelSet ... completed in 5 seconds and 110.9 milliseconds (5110.9)
ParticlesToLevelSet ... completed in 5 seconds and 149.0 milliseconds (5149.0)
ParticlesToLevelSet ... completed in 5 seconds and 329.7 milliseconds (5329.7)

With the exception of the first run (which unlike yours was the slowest) the subsequent runs all take approximately the same time (within the accuracy of timing on a system running other processes in the background). Since I don't have access to a Windows machine I can't help you beyond saying this is not a known issue under Linux. Maybe someone else on this forum can try to reproduce your observation on Windows ...

Cheers,
Ken

[edward]

On Tuesday, February 6, 2018 at 3:45:07 AM UTC-5, Ken Museth wrote:

Since I don't have access to a Windows machine I can't help you beyond saying this is not a known issue under Linux. Maybe someone else on this forum can try to reproduce your observation on Windows ...

I could give it a try, but ONLY if there was a reproducible self-contained example posted on this list. This problem smells like a classic memory fragmentation so very likely heap allocator dependent, which means the answer might be "you're building it wrong". :)

-Edward