[eq-dev] Distributed Scenegraph - Memory Leak

6 views
Skip to first unread message

Matteo Hausner

unread,
May 3, 2012, 4:21:26 PM5/3/12
to eq-...@equalizergraphics.com
Hi everybody,

First of all I'm relatively new in developing with Equalizer. As a
student at the university of Kempten (Germany), I'm currently working
on a project to create an application running on a cluster, consisting
of four workstations, each harbouring two Nvidia Quadro Cards. This
cluster is used to drive a CAVE consisting of three walls, with two
beamers for each wall, to achieve passive stereoscopic rendering.

The goal of the whole project is to develop a simple game on top of an
already existing scenegraph, that has to be integrated into Equalizer.

So far integrating the scenegraph has run pretty well - but now I have
reached a point, where I could use some advice...
Generally my implementation is using the concept of dynamic data distribution.
A few days ago I noticed that my current implementation is running,
but has some severe memory leaks:
Each frame the application is leaking a huge amount of memory, and
even more of it if I'm doing transformations on scenegraph nodes or
dynamically add more nodes to the graph.

Currently the root node of my graph, distDSG::DistNode resides in the
FrameData object derived from co::Serializable.
All the scenegraph nodes themselves, are implementing co::Serializable
as well. (DELTA changetype)
When adding a child node to a parent, I set an appropriate DirtyBit
(DIRTY_CHILDREN) in the parent node.
In distDSG::DistNode::serialize() I serialize the ID of both children
and in distDSGNode::deserialize(), I create a new instance of the
appropriate class (via an also serialized class identifier) and map it
to the ID:
...
LeftChild = getInstanceOf(leftChildClass);
localNode->mapObject(LeftChild, leftChildID);
...

I'm manipulating the graph via eq::Config, through functions mapped to
keyboard shortcuts.
After manipulating or adding a node of the graph, I set the
appropriate DirtyBits of the the node itself, as well DirtyBits on all
of it's parents up to the root node, that mark a node that has dirty
children.
In eq::Config::startFrame() commit() gets called on the FrameData object.
Calling commit() on FrameData or a node of the graph, automatically
passes the call on to all children, so that every node gets committed
for each frame. On calling commit I store the version value for the
root node inside the FrameData object and the versions for a the
children of a node inside the parent-node itself.

Now in eq::Channel::frameDraw() I draw the graph using a simple class
called Renderer, which is member of eq::Window, calling
Renderer->Render() invokes sync(frameID) on the root node and
afterwards the traversal of the graph.
Similarly to the commit() call that gets passed on, I pass on the
sync() call through the whole graph, using the version values stored
inside the parent nodes.

Generally my implementation seems to be working, but in its current
state it is leaking those huge amounts of memory every frame, after
adding nodes to the graph and even more, when performing
transformations onto graph nodes.
I'm pretty sure that I got something wrong in the way manipulations on
nodes are distributed and synchronized, but even after days of testing
I couldn't find a solution to my problem.

The project is still running until the month July and I already spoke
to the other people involved, that once we have reached a final and
working state we might be willing to hand our source out, to serve as
an example application. Unfortunately right now there seems to be no
freely available implementation of a dynamic scenegraph based on
Equalizer(?).

Anyways I would greatly appreciate if someone could point me towards
the cause of the described issue...

Thanks and best regards,
Matteo

_______________________________________________
eq-dev mailing list
eq-...@equalizergraphics.com
http://www.equalizergraphics.com/cgi-bin/mailman/listinfo/eq-dev
http://www.equalizergraphics.com

Stefan Eilemann

unread,
May 4, 2012, 4:41:53 AM5/4/12
to Equalizer Developer List
Hi Matteo,

On 3. May 2012, at 22:21, Matteo Hausner wrote:

> The goal of the whole project is to develop a simple game on top of an
> already existing scenegraph, that has to be integrated into Equalizer.

Nice. :)

> [snip]

So far it looks like one typical way to implement scene graph distribution.

> Now in eq::Channel::frameDraw() I draw the graph using a simple class
> called Renderer, which is member of eq::Window, calling
> Renderer->Render() invokes sync(frameID) on the root node and
> afterwards the traversal of the graph.
> Similarly to the commit() call that gets passed on, I pass on the
> sync() call through the whole graph, using the version values stored
> inside the parent nodes.

I'm assuming you've got one copy of the SG per node. In that case, you should do the sync() in Node::frameStart().

> Generally my implementation seems to be working, but in its current
> state it is leaking those huge amounts of memory every frame, after
> adding nodes to the graph and even more, when performing
> transformations onto graph nodes.

Is the application, the render clients or both leaking?

You've got to figure out who is leaking the memory. The delta objects buffer quite some stuff (see [1]), so that might be one cause. However, they should not keep leaking memory over time.

Have a try with valgrind on a small scene, let it run for some frames with modifications and see if you have hard losses, or if it's simply a matter of data pooling up somewhere.

> The project is still running until the month July and I already spoke
> to the other people involved, that once we have reached a final and
> working state we might be willing to hand our source out, to serve as
> an example application. Unfortunately right now there seems to be no
> freely available implementation of a dynamic scenegraph based on
> Equalizer(?).

That's unfortunately correct, and this project would indeed be interesting. What SG are we talking about?


HTH,

Stefan.

Stefan Eilemann

unread,
May 4, 2012, 9:53:05 AM5/4/12
to Equalizer Developer List
Hi,

one more thing: having a simple reproducer in eqPly or standalone will help you to isolate the problem and me to give you advice or fix the bug.


Cheers,

Stefan.

Matteo Hausner

unread,
May 4, 2012, 11:18:45 AM5/4/12
to Equalizer Developer List
Hello Stefan,

thanks for your quick reply!

2012/5/4 Stefan Eilemann <eile...@gmail.com>:
> I'm assuming you've got one copy of the SG per node. In that case, you should do the sync() in Node::frameStart().

That makes sense, so I moved the sync() call from the Renderer to
Node::frameStart().

> Is the application, the render clients or both leaking?

There was also a leak inside the application, which I tracked down by now.

> You've got to figure out who is leaking the memory. The delta objects buffer quite some stuff (see [1]), so that might be one cause. However, they should not keep leaking memory over time.

This application leak was the cause for the continuous memory leaking
with every frame.
But the issue of increased memory usage, each time I modify a
scenegraph node still exists.
I lose around 150 to 300 kb of memory each time I conduct a
transformation on the graph, or even just change the name (string) of
a node.

> Have a try with valgrind on a small scene, let it run for some frames with modifications and see if you have hard losses, or if it's simply a matter of data pooling up somewhere.

Unfortunately one requirement for the project was to run on Windows,
so I decided to start development on a Windows VM, as it will be our
target OS on the CAVE.
I tried several other profiling tools (Deleaker, Visual Leak Detector,
Very Sleepy), that are available for the Windows platform, but I
haven't been able
to narrow down how this rather huge amounts of memory are
accumulating. However I noticed that a major fraction of the CPU time
(> 80%) is being spent a kernel32 function WaitForMultipleObjects()
called from co::Global::setObjectBufferSize(). As I did not notice
this behavior with eqPly, I'm assuming this might be part of the
increasing memory footprint issue...(?)

> That's unfortunately correct, and this project would indeed be interesting. What SG are we talking about?

The scenegraph we use was a personal project of the professor
supervising the whole CAVE project. It was mainly developed to provide
an easy to use scenegraph for students starting to work with OpenGL.
However it offers a pretty solid collection of node types, up to the
integration of shaders and FBOs.

Thanks and best regards,
Matteo

Stefan Eilemann

unread,
May 4, 2012, 12:44:06 PM5/4/12
to eq-...@equalizergraphics.com

On 4. May 2012, at 17:19, Matteo Hausner [via Software] wrote:

> There was also a leak inside the application, which I tracked down by now.

So I assume that means the render clients are leaking. Most likely this is due to missing sync() on some objects. Try setting the 100 to 10 in Object::notifyNewHeadVersion and see who is throwing an assertion.


HTH,

Stefan.

--
View this message in context: http://software.1713.n2.nabble.com/Distributed-Scenegraph-Memory-Leak-tp7525052p7527762.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.

Matteo Hausner

unread,
May 6, 2012, 11:11:04 AM5/6/12
to Equalizer Developer List
Hi Stefan,

I think you got me on the right track there:

> So I assume that means the render clients are leaking. Most likely this is due to missing sync() on some objects. Try setting the 100 to 10 in Object::notifyNewHeadVersion and see who is throwing an assertion.

After setting the value to 10, I'm getting an assertion on the root
node object, which I'm keeping in the FrameData. It looks like all the
previous versions are still kept alive until the assertion is thrown.
I'm pretty sure this happens when deserializing a graph node object,
so for reference here's my implementation for a general graph node,
like the root element:

void DistNode::deserialize(co::DataIStream &is, const uint64_t dirtyBits)
{
DistSerializable::deserialize( is, dirtyBits );

if( dirtyBits & DIRTY_CHILDREN )
{
// Obtain Class and ID of LeftChild
Class leftChildClass;
is >> leftChildClass;
co::base::UUID leftChildID;
is >> leftChildID;
is >> _leftChildVersion;

// Obtain Class and ID of RightSibling
Class rightSiblingClass;
is >> rightSiblingClass;
co::base::UUID rightSiblingID;
is >> rightSiblingID;
is >> _rightSiblingVersion;

// Instantiate objects for LeftChild and Right Sibling
// And map LeftChild and RightSibling to master ID
co::LocalNodePtr localNode = getLocalNode();

if(leftChildID != co::base::UUID(0))
{
DistNode *newLeftChild = getInstanceOf(leftChildClass);
localNode->mapObject(newLeftChild, leftChildID);
if(LeftChild)
if(LeftChild->isAttached())
{
localNode->releaseObject(LeftChild);
}
LeftChild = newLeftChild;
}
else
LeftChild = NULL;

if(rightSiblingID != co::base::UUID(0))
{
DistNode *newRightSibling = getInstanceOf(rightSiblingClass);
localNode->mapObject(newRightSibling, rightSiblingID);
if(RightSibling)
if(RightSibling->isAttached())
{
localNode->releaseObject(RightSibling);
}
RightSibling = newRightSibling;
}
else
RightSibling = NULL;
}
if( dirtyBits & DIRTY_NAME )
{
is >> _name;
}
}

For testing purpose I added the releaseObject() call. However I'm not
really sure if this is really needed, or if something else is missing
here...
I still seem to be somewhat lost... Is there anything else I have to
take care of before I map my new instances, or does Equalizer manage
everything by itself?

Thanks for all your input so far, you have been a great help Stefan. :)

Cheers,
Matteo

Stefan Eilemann

unread,
May 13, 2012, 3:52:40 AM5/13/12
to Equalizer Developer List

On 6. May 2012, at 17:11, Matteo Hausner wrote:

>> So I assume that means the render clients are leaking. Most likely this is due to missing sync() on some objects. Try setting the 100 to 10 in Object::notifyNewHeadVersion and see who is throwing an assertion.
>
> After setting the value to 10, I'm getting an assertion on the root
> node object, which I'm keeping in the FrameData. It looks like all the
> previous versions are still kept alive until the assertion is thrown.

It means that you have a slave object which is behind calling sync() for more than ten versions. This typically happens if you forget to sync() your slave instances on the render clients, either because you're missing to traverse to the object or you've abandoned, but not unmapped and deleted the object.

> For testing purpose I added the releaseObject() call. However I'm not
> really sure if this is really needed, or if something else is missing
> here...
> I still seem to be somewhat lost... Is there anything else I have to
> take care of before I map my new instances, or does Equalizer manage
> everything by itself?

If you've had a previous object which you are going to abandon, you have to unmap/release it. The design looks somewhat fishy, since you're recreating children objects when a child changes? I would expect this to happen only if the child itself is a different object.


HTH,

Stefan.

Matteo Hausner

unread,
May 16, 2012, 1:38:12 PM5/16/12
to Equalizer Developer List
Hi Stefan,

thanks for your reply!

2012/5/13 Stefan Eilemann <eile...@gmail.com>:
> If you've had a previous object which you are going to abandon, you have to unmap/release it. The design looks somewhat fishy, since you're recreating children objects when a child changes? I would expect this to happen only if the child itself is a different object.

You are right, I guess I didn't really think this through...
This must also be the reason why manipulations on graph nodes are only
passed on, if I set the dirty bits for children on all nodes residing
above them in the graph.
However I can't really figure out a way to implement a correct
synchronization behavior, without creating a new node instance every
time.

For reference this is how my commit() / syc() implementation currently
looks like in the DistNode class:
co::uint128_t DistNode::commit( const uint32_t incarnation)
{
if(LeftChild)
{
_leftChildVersion = LeftChild->commit(incarnation);
}
if(RightSibling)
{
_rightSiblingVersion = RightSibling->commit(incarnation);
}

return co::Serializable::commit(incarnation);
}

co::uint128_t DistNode::sync( const co::uint128_t& version)
{

if(LeftChild)
LeftChild->sync(_leftChildVersion);
if(RightSibling)
RightSibling->sync(_rightSiblingVersion);

return co::Serializable::sync(version);
}

This approach seems to be the reason why I ran into this problem in
the first place.
I would be super happy, if you could give me some more advice on how
to implement a correct commit / synchronization behavior for a
distributed graph in Equalizer...

Thanks and best regards,
Matteo

Stefan Eilemann

unread,
May 17, 2012, 5:31:53 AM5/17/12
to Equalizer Developer List

On 16. May 2012, at 19:38, Matteo Hausner wrote:

>> If you've had a previous object which you are going to abandon, you have to unmap/release it. The design looks somewhat fishy, since you're recreating children objects when a child changes? I would expect this to happen only if the child itself is a different object.
>
> You are right, I guess I didn't really think this through...
> This must also be the reason why manipulations on graph nodes are only
> passed on, if I set the dirty bits for children on all nodes residing
> above them in the graph.
> However I can't really figure out a way to implement a correct
> synchronization behavior, without creating a new node instance every
> time.

The general logic should be to sync() the child if only the version changed, and to create a new one if the id changed. The full code might be a bit hairy at the end, depending on you implementation details.

> I would be super happy, if you could give me some more advice on how
> to implement a correct commit / synchronization behavior for a
> distributed graph in Equalizer...

The details you'll have to decide on yourself, I'm afraid. It's always a bit different, depending on your use case. The co::DataO/IStream::(de)serializeChildren might help as an inspiration.


HTH,

Stefan.

Matteo Hausner

unread,
May 20, 2012, 12:27:20 PM5/20/12
to Equalizer Developer List
Hello Stefan,

I could finally get rid of this problem.
Again thanks a lot for all your input on this matter! :-)

Best regards,
Matteo

Matteo Hausner

unread,
Jun 18, 2012, 3:16:14 PM6/18/12
to Equalizer Developer List
Hi Stefan,

our project is nearing completion but still there are two issues
remaining on which I could use your guidance:

First of all I still have some issues when deleting SG-Nodes from my graph.
I need to know when the right order of calls I have to do in order to
get rid of an existing versioned distributed object.
Currently I'm calling localNode->unmap(myObject) on the
ApplicationNode first and delete myObject on the Rendering-Node,
however when calling delete on the object I'm receving something like:
"Debug Assertion Failed! ... Expression: _BLOCK_TYPE_IS_VALID(
pHead->nBlockUse )".

The second issue I'm having is about setting the focal point for our
passive stereo setup.
When trying to set the focal point via the config-file I'm getting a
"Unimplemented code" error. Is there any documentation, on how to set
a fixed focal point?

Thanks already and best regards,

Stefan Eilemann

unread,
Jun 19, 2012, 2:56:02 AM6/19/12
to eq-...@equalizergraphics.com
Mi Matteo,

On 18. Jun 2012, at 21:17, Matteo Hausner [via Software] wrote:

> First of all I still have some issues when deleting SG-Nodes from my graph.
> I need to know when the right order of calls I have to do in order to
> get rid of an existing versioned distributed object.

*Ideally* you do:

First on render clients: unmap, delete
Then on app Node: deregister, delete

If you can't guarantee the order between render clients and app, that's not too bad since the master instance will forcefully unmap all clients.

> Currently I'm calling localNode->unmap(myObject) on the
> ApplicationNode first and delete myObject on the Rendering-Node,
> however when calling delete on the object I'm receving something like:
> "Debug Assertion Failed! ... Expression: _BLOCK_TYPE_IS_VALID(
> pHead->nBlockUse )".

That's a heap corruption which can be caused by anything. There have been reports that the 1.2 binaries don't work for some users - do you use these or did you compile Eq by yourself?

Also, the stack trace might give some hints at what's going on.

> The second issue I'm having is about setting the focal point for our
> passive stereo setup.
> When trying to set the focal point via the config-file I'm getting a
> "Unimplemented code" error. Is there any documentation, on how to set
> a fixed focal point?

http://www.equalizergraphics.com/documents/design/immersive.html

What are you setting, and which unimplemented code are you hitting?


HTH,

Stefan.



--
View this message in context: http://software.1713.n2.nabble.com/Distributed-Scenegraph-Memory-Leak-tp7525052p7580537.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.

Reply all
Reply to author
Forward
0 new messages