[osg-users] PagedLOD experts?
Chris 'Xenon' Hanson
obviously) who have delved into the code for PagedLOD and DatabasePager and actually done
any coding in there?
--
Chris 'Xenon' Hanson, omo sanza lettere Xenon AlphaPixel.com
PixelSense Landsat processing now available! http://www.alphapixel.com/demos/
"There is no Truth. There is only Perception. To Perceive is to Exist." - Xen
_______________________________________________
osg-users mailing list
osg-...@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
sergey leontyev
i dont know any, but since you are one of those few "elite" members of the OSG community :-) Maybe you can help me understand what is going on:
http://forum.openscenegraph.org/viewtopic.php?t=3782
Thanks!
Sergey
Chris 'Xenon' Hanson wrote:
> I'm just wondering -- other than myself, are there any others (excluding Robert,
> obviously) who have delved into the code for PagedLOD and DatabasePager and actually done
> any coding in there?
>
> --
> Chris 'Xenon' Hanson, omo sanza lettere Xenon AlphaPixel.com
> PixelSense Landsat processing now available! http://www.alphapixel.com/demos/
> "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen
> _______________________________________________
> osg-users mailing list
>
> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
>
> ------------------
> Post generated by Mail2Forum
------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=19000#19000
Wang Rui
Paul Martz
client databases. But I haven't done any coding there. I'm familiar with
the environment variables that control its behavior, what operations are
done at what phase of the frame, and am aware of how the PagedLOD class
differs significantly from its base class.
In the vast majority of my current work, I'm not using the DatabasePager
or PagedLOD classes at all, so am probably a bit rusty on the current
code base.
Paul Martz
Skew Matrix Software LLC
_http://www.skew-matrix.com_ <http://www.skew-matrix.com/>
+1 303 859 9466
Chris 'Xenon' Hanson
> Chris,
> i dont know any, but since you are one of those few "elite" members of the OSG community :-) Maybe you can help me understand what is going on:
> http://forum.openscenegraph.org/viewtopic.php?t=3782
I can't tell you much about TXP paging or multi-camera Database paging.
What I can tell you is you need to do some in-situ debugging. You need to compile with
debugging and use memory debugging tools to see what sort of objects (what size of
allocations) are being allocated. Add code wherever needed to count these allocations, and
set breakpoints to see where they're being allocated from.
Turn on as much OSG debugging info (OSG Notify level DEBUG_INFO) and watch what spits
out. Usually OSG tells you what it's doing in detail if it's loading or allocating stuff.
Add more Notify debugging to suspicious code areas if you need to.
I could probably help you more with this, but I've got to help my paying client.
> Thanks!
> Sergey
--
Chris 'Xenon' Hanson, omo sanza lettere Xenon AlphaPixel.com
PixelSense Landsat processing now available! http://www.alphapixel.com/demos/
"There is no Truth. There is only Perception. To Perceive is to Exist." - Xen
_______________________________________________
osg-users mailing list
osg-...@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Chris 'Xenon' Hanson
> I'm just wondering -- other than myself, are there any others (excluding Robert,
> obviously) who have delved into the code for PagedLOD and DatabasePager and actually done
> any coding in there?
I'll throw out an item that I think should be discussed from my recent work:
DatabasePager.cpp, capped_removeExpiredSubgraphs() has a line of coding that says:
MarkPagedLODsVisitor markerVistor("NeedToRemove");
This visitor, when applied, goes through the _inactivePagedLODList, and renames a
certain number of the contents to "NeedToRemove". Later, if you are NOT using the
_deleteRemovedSubgraphsInDatabaseThread option, it removes PagedLOD nodes that have this name.
This seems to me to be a dangerous usage of the node name. Granted, it's very unlikely
that you'd have a node named "NeedToRemove", but it underscores that the practice of
assigning special meaning to the node name isn't especially safe. I think it would be wise
to clean the node names as the nodes are added to the _inactivePagedLODList so that none
of them could ever possibly enter into the list already bearing the name "NeedToRemove".
What I haven't figured out is if there's ever a situation where a node _leaves_ the
inactive list and returns to the active scenegraph without being deleted. In this case,
any tampering of the node name would have undesirable results.
It does raise another question -- is this a common practice and is there anywhere else
in OSG where node name has a special meaning during specific phases of operation? This
particular one is not documented anywhere I know of. In this case, the failure mode would
be pretty benign -- the DatabasePager could accidentally destroy too many inactive
PagedLOD nodes in one frame-idle period. Normally this is constrained by the
_targetMaximumNumberOfPageLOD. So, maybe this failure could cause the redraw to break
framerate. Nothing fatal.
And finally, is there a better extensible way to go about marking arbitrary nodes for
later processing (while potentially still in the scenegraph, even though this case has
them removed from the scenegraph by this point)? Any fast boolean flag will be immediately
overwhelmed because everyone will want to use it and they'll collide. But any extensible
method would seem to involve allocating and attaching some sort of dynamic marker, which
is not especially efficient.
Any thoughts?
Tim Moore
> I'm just wondering -- other than myself, are there any others (excluding Robert,
> obviously) who have delved into the code for PagedLOD and DatabasePager and actually done
> any coding in there?
>
order to do paging without using PagedLOD.
Tim
Wang Rui
This visitor, when applied, goes through the _inactivePagedLODList, and renames a
certain number of the contents to "NeedToRemove". Later, if you are NOT using the
_deleteRemovedSubgraphsInDatabaseThread option, it removes PagedLOD nodes that have this name.
It does raise another question -- is this a common practice and is there anywhere else
in OSG where node name has a special meaning during specific phases of operation?
And finally, is there a better extensible way to go about marking arbitrary nodes for
later processing (while potentially still in the scenegraph, even though this case has
them removed from the scenegraph by this point)? Any fast boolean flag will be immediately
overwhelmed because everyone will want to use it and they'll collide. But any extensible
method would seem to involve allocating and attaching some sort of dynamic marker, which
is not especially efficient.
Any thoughts?
Jean-Sébastien Guay
> It does raise another question -- is this a common practice and is there anywhere else
> in OSG where node name has a special meaning during specific phases of operation?
I tend to think that any use of user-settable properties, such as node
name, callbacks, user data, etc. inside OSG is potentially dangerous.
Any app could potentially change the node's name, remove/replace a
callback or user data and that would break the library's behavior, and
the only way to know why it's broken is to go delve into OSG's source.
For callbacks, it's pretty easy to instead derive a node type and
override the traverse() method. That way it's much harder for the user
to break expected behavior of the node (they would need to derive from
this node type, override traverse() and forget to call the base class
version). For name and user data, I don't see an easy way to avoid using
them if OSG already uses them in some cases... But in any case,
identifying nodes by name always strikes me as unsafe.
Since we use visitors so often in OSG coding, OSG itself can't assume
that any node is hidden from the user, and so can't assume that
user-settable properties won't be changed by the user...
This doesn't really help you, but I wanted to be more detailed than just
saying "I agree".
J-S
--
______________________________________________________
Jean-Sebastien Guay jean-seba...@cm-labs.com
http://www.cm-labs.com/
http://whitestar02.webhop.org/
Chris 'Xenon' Hanson
> This doesn't really help you, but I wanted to be more detailed than just
> saying "I agree".
Thanks for th insight, everyone.
I'm kind of an old-school C-flavor programmer, so my instinctive response is rather
C-like. I don't like bloating people's in-memory objects, so I'd lean towards defining a
32-bit flag member on class Node called something like _ephemeralInternalStateFlags. This
would not be saved or restored, and the definitions of each flag bit would be enumerated
as 1<<n enums in the Node class header. Code outside of the OSG core distribution would be
prohibited (by policy, can't really enforce it in the compiler) from defining and
utilizing these flag bits, but code within OSG core could. The first bit allocated would
be used to replace the NeedToRemove indicator.
Also, while the read-modify-write cycle used to toggle just one bit of a flag is not as
fast as the blind-write cycle used when you dedicate a whole boolean variable (and consume
more memory), both the setting operation AND the testing operation should be faster than
the current string-marking technique.
If Robert has no objections to this solution, I'll code it up quick and share it.
I'm also concerned about the amount of very similar-but-not-quite-the-same code between
DatabasePager::capped_removeExpiredSubgraphs and
DatabasePager::expiry_removeExpiredSubgraphs. I'd like to refactor that to DRY (Don't
Repeat Yourself) it out a bit as I'm implementing some code of my own that needs to
cleanly dispose of some PagedLOD children. But, I don't understand why there are
differences in the disposal technique between them (one calls
_fileRequestQueue->updateBlock() directly, one sets a local flag and calls updateBlock()
at the end of the operation, etc).
> J-S
--
Chris 'Xenon' Hanson, omo sanza lettere Xenon AlphaPixel.com
PixelSense Landsat processing now available! http://www.alphapixel.com/demos/
"There is no Truth. There is only Perception. To Perceive is to Exist." - Xen
Robert Osfield
I just can't contribute to complex threads right now as I need all my
interlect to get the GLES/GL port right, so I won't even try to chip
in. Please just be patient and hang back from hacking code that
requires my input on to get a full understanding.
Robert.
Paul Martz
> Jean-Sébastien Guay wrote:
>> This doesn't really help you, but I wanted to be more detailed than just
>> saying "I agree".
>
> Thanks for th insight, everyone.
>
> I'm kind of an old-school C-flavor programmer, so my instinctive response is rather
> C-like. I don't like bloating people's in-memory objects, so I'd lean towards defining a
> 32-bit flag member on class Node called something like _ephemeralInternalStateFlags. This
> would not be saved or restored, and the definitions of each flag bit would be enumerated
> as 1<<n enums in the Node class header.
Oh, excellent! Then you can join the debate. See: "Change to Optimizer
OptimizationOptions" or (in osg-submissions) "DisplaySettings masks for
implicit buffer attachments".
:-)
-Paul
Paweł Księżopolski
> where a node _leaves_ the inactive list and returns to the
> active scenegraph without being deleted. In this case,
> any tampering of the node name would have undesirable results.
There is at least one scenario in which PagedLODs named "NeedToRemove"
go to active list. It happens when you use
Registry::instance()->_objectCache.
The scenario looks like this:
- PagedLODs marked as "NeedToRemove" are removed from _inactivePagedLODList.
They also go to _fileRequestQueue->_childrenToDeleteList.
- request queue releases these PagedLODs, but they are still in
Registry::instance()->_objectCache.
- _objectCache will release them in 10 seconds if no one will request
for them.
- if databasePager will request using them again, FindPagedLODsVisitor
will place these PagedLODs
on _activePagedLODList ( it's important to mention that these
PagedLODs currently keep
"NeedToRemove" name )
- sooner or later these PagedLODs will go to _inactivePagedLODList.
Suppose we have 4 such PagedLODs( let's name them : A,B,C,D ).
- they will not be removed because variable numToRemove==0
- now, databasePager wants to remove another 4 PagedLODs ( marks them as
"NeedToRemove",
and places in _fileRequestQueue->_childrenToDeleteList ). Let's name those
removed PagedLODs as : E,F,G,H.
- At that moment we have 8 PagedLODs marked as "NeedToRemove" on
_inactivePagedLODList,
but only 4 of them will be removed from _inactivePagedLODList because
databasePager
has variable numToRemove==4( probably A,B,C and D - thet're first on
_inactivePagedLODList.
But they probably shouldn't be removed now - their "NeedToRemove" name
is here by accident ).
This variable holds the quantity of PagedLods named "NeedToRemove" in
last iteration ( E,F,G,H ).
- The quantity of PagedLODs named "NeedToRemove" in
_inactivePagedLODList will raise
until application takes all available memory.
- At that moment it is also possible that inactive PagedLODs named
"NeedToRemove" go directly
to active list if required ( and vice versa. See statistics below ).
In order to remove this error we've done two fixes :
1. We changed method DatabasePager::FindPagedLODsVisitor::apply() :
virtual void apply(osg::PagedLOD& plod)
{
plod.setFrameNumberOfLastTraversal(_frameNumber);
if(plod.getName()=="NeedToRemove")
plod.setName("");
_activePagedLODList.push_back(&plod);
traverse(plod);
}
2. We removed condition numRemoved<numToRemove, when PagedLODs named
"NeedToRemove" are
removed from _inactivePagedLODList:
for(PagedLODList::iterator itr = _inactivePagedLODList.begin();
itr!=_inactivePagedLODList.end() /*&& numRemoved<numToRemove*/; )
The second fix destroys optimization and is probably not necessary,
because, we haven't
observed PagedLODs named "NeedToRemove", that go from
_activePagedLODList to _inactivePagedLODList
after we used first fix.
At the end, we can show you some statistics for our terrain database -
we where flying over the terrain
using the same AnimationPath:
1. Our application using original SVN source :
See attached image PagedLOD_svn.png.
Column B is the sum of PagedLODs (active+inactive), Column C - active
PagedLODs,
column D - inactive PagedLODs.
As you can see - the count of inactive PagedLODs raises rapidly.
The count of active PagedLODs moves chaotically between 150 and 300
PagedLODs.
Movement NTR / other names
FindPagedLODsVisitor->_activePagedLODList : 1008 / 3600
_activePagedLODList->_inactivePagedLODList : 7358 / 12897
_inactivePagedLODList->_activePagedLODList : 6101 / 8367
_inactivePagedLODList->_objectCache : 3232 / 3232
We see heavy traffic between active and inactive list. Most of
the PagedLODs moving from active to inactive ( and vice versa )
has a name "NeedToRemove".
2. Our application using source with both fixes:
See attached image PagedLOD_2_fixes.png.
The Description of columns is the same as above.
Sum of active and inactive PagedLODs is constant.
The only thing that bothers me is the count of inactive PagedLODs.
Movement NTR / other names
FindPagedLODsVisitor->_activePagedLODList : didnt test it
_activePagedLODList->_inactivePagedLODList : 0 / 9583
_inactivePagedLODList->_activePagedLODList : 0 / 5189
_inactivePagedLODList->_objectCache : 3992/3992
No PagedLODs with name "NeedToRemove" goes from active to inactive
or vice versa.
3. Our application using only first fix :
See attached image PagedLOD_1_fix.png.
The Description of columns is the same as above.
Sum of active and inactive PagedLODs is constant.
Count of inactive PagedLODs is lower than in second example.
Movement NTR / other names
FindPagedLODsVisitor->_activePagedLODList : 0/4584
_activePagedLODList->_inactivePagedLODList : 0 / 9207
_inactivePagedLODList->_activePagedLODList : 0 / 4609
_inactivePagedLODList->_objectCache : 4276/4276
Example 2 and 3 look similar ( except for inactive to active movement,
i dont know where is this difference coming from - need additional
digging ),
so there is no need to use second fix. Optimization is saved :)
--
Paweł Księżopolski
Robert Osfield
On Mon, Nov 2, 2009 at 4:05 PM, Chris 'Xenon' Hanson
<xe...@alphapixel.com> wrote:
> I'm kind of an old-school C-flavor programmer, so my instinctive response is rather
> C-like. I don't like bloating people's in-memory objects, so I'd lean towards defining a
> 32-bit flag member on class Node called something like _ephemeralInternalStateFlags. This
> would not be saved or restored, and the definitions of each flag bit would be enumerated
> as 1<<n enums in the Node class header. Code outside of the OSG core distribution would be
> prohibited (by policy, can't really enforce it in the compiler) from defining and
> utilizing these flag bits, but code within OSG core could. The first bit allocated would
> be used to replace the NeedToRemove indicator.
I'm not sure whether this is really needed. The NeedToRemove trick in
DatabasePager is very specific to a piece of code embedded in
DatabasePager it has no value outside this specific bit of code, and I
believe it's not appropriate to build algorithm specific such support
into the core OSG.
> Also, while the read-modify-write cycle used to toggle just one bit of a flag is not as
> fast as the blind-write cycle used when you dedicate a whole boolean variable (and consume
> more memory), both the setting operation AND the testing operation should be faster than
> the current string-marking technique.
>
> If Robert has no objections to this solution, I'll code it up quick and share it.
I think the right thing to do is nothing in the core OSG, and look at
the specific DatabasePager algorithm to retain the O(1) cost of the
check that I was using, with the bugs with handling multiple
viewpoints.
> I'm also concerned about the amount of very similar-but-not-quite-the-same code between
> DatabasePager::capped_removeExpiredSubgraphs and
> DatabasePager::expiry_removeExpiredSubgraphs. I'd like to refactor that to DRY (Don't
> Repeat Yourself) it out a bit as I'm implementing some code of my own that needs to
> cleanly dispose of some PagedLOD children. But, I don't understand why there are
> differences in the disposal technique between them (one calls
> _fileRequestQueue->updateBlock() directly, one sets a local flag and calls updateBlock()
> at the end of the operation, etc).
Both methods exists while we test the new
capped_removeExpiredSubgraphd , once it's fully tested and debugged
the old expiry_removeExpiredSubgraphs method will be removed
completely this neatly avoiding the issue of similar code.
Robert.
Wojciech Lewandowski
Robert Osfield
On Mon, Nov 23, 2009 at 11:53 AM, Wojciech Lewandowski
<lewan...@ai.com.pl> wrote:
> 3. I don't know if this was deliberate or not, but IntersectionVisitor
> in USE_HGHEST_LEVEL_OF_DETAIL mode does load highest level tiles temporary.
> It does intersections and free them. It does not hook them up to their
> parents.
This is deliberate - if you start modifying the scene graph during the
Intersection traversal then you can't run it multi-threaded anymore.
There is also the issue of having to maintain a PageLOD with it's
children loaded from lowest res to highest res - you can't skip
intermediate children and just attach the highest rest one, keeping
the loaded subgraph local.
It's also worth mentioning that IntersectionVsitor itself does do any
loaded of data, it's callback that you attach to it that do it so
usual usage of IntersectionVisitor doesn't load highest level of
detail - it doesn't load anything. Hence the
DatabaseCacheReadCallback...
> 4. Above has this horrible effect that if some node was not already loaded
> by DatabasePager it will be constantly loaded and removed by
> IntersectionVisitor. SARCASM#1: But don't loose faith there is a solution.
> See next point.
Well if you want the IntersectionVisitor to happen asynchronously from
the DatabasePager loaded then you have to have keep the two separate,
or manage the integration very careful by using a
IntersectionReadCallback that is design to integrate the two.
Please note DatabaseCacheReadCallback is not designed to integrate
DatabasePager and IntersectionVisitor, it's a specific solution for
IntersectionVisition and is wholly parallel and independent to
DatabasePager and is meant to be this way. IntersectionVisitor is
synchronous - you have to get the subgraph immediately, so the reading
of external tiles has to be done in the same thread.
> 5. We found out that we can use osgSim::DatabaseCacheReadCallback to
> mitigate former problem. This is an IntersectVisitor read callback that
> keeps internal cache to avoid repetitive loading and freeing. Well...this
> works but only for IntesectVisitor. DatabasePager does not know anything
> about this cache so when DatabasePager finally decides to load a tile it
> does it again, although IntersectionVisitor have already loaded it.
> SARCASM#2: But don't loose hope yet, because we found a solution to this as
> well....
> 6. How to make sure DatabasePager sees the tile IntersectVisitor already
> loaded ? Its simple: We could use osgDB cache. So we started to load tiles
> with CACHE_NODES option and everything seemed to be fine....
> 7 But after some time we started to observe crashes and memory leaks. Long
> story about them is in my friend Pawel Ksiezopolski post "Re: [osg-users]
> PagedLOD experts?" from November 5th. Short story is that caching tiles
> does not free renamed "NeedToRemove" nodes, but keep them in memory so with
> time some if them land int the scene again. When this happens, logic that
> was invented to remove not used nodes removes wrong ones leaving those that
> should be removed. Hence we get PageLOD thrashing and memory leaks. Cool HUH
> ? (Yes its SARCASM#3)
Well if you do start mixing stuff that wasn't intended to be mixed in
the way you are mixing it well perhaps we shouldn't be too surprised
that problems eventually do ensue.
Also the SCARCASM stuff really isn't helpful and just lowers to tone
of communication. Please stop it, it has no place on osg-users. If
you have a technical point to make it without the extra crap.
> Now to conclusion:
> In my opinion PageLOD renaming is the reason our elaborated scheme failed. I
> am both unhappy and glad that it happened because it clearly shows that
> Object::_name should not be modified by internal OSG methods. It should be
> never ever changed by OSG. Management of node name belongs to creators or
> users of the node.
Um.... your elaborate scheme illustrates a weakness in the flexibility
of the performance trick in DatabasePager, not anything about using
Object::_name. It also illustrates problems your elaborate scheme.
The performance trick in DatabasePager is a bug because it just is
robust in the presence of multiple-viewpoints and mixing of threading
of loading.
> Thats why I am going to prepare a fix that will not rename nodes to remove.
> But will instead drop their addresses into a set (or sorted vector) and
> will later us this set to test if node is marked for removal. It may be not
> O(n) but O(n log n) but it will work at least. Is anyone preparing
> similar fix ?
Ironically, I originally did implement the code with a std::set<> but
went for using setName/getName() to make the code more efficient. In
hindsight this optimization this may have been a mistake, but until
I've actually recreated the multi-viewpoint bug and dug down into
exactly what was going wrong I can't say for sure where just replace
it with std::set will fix all the problems.
In your case you will need to think more deeply about mixing
IntersectionVisitor and DatabasePager loading. There aren't meant to
be mixed.
Wojciech Lewandowski
My apologies for "SARCASM". I wanted to be provocative because former posts
went unnoticed. Maybe I have gone too far...
But I would strongly defend merits of arguments in the post. You say we did
wrong, but whats you recommendation on mixing many intersections with
rendering ? Its very common scenario. How should we tackle it, if current
approach is wrong ?
Second important issue for me is usage of _name in scene graph. I always
expected that _name is reserved for users and its a normal rule in all Scene
Graph implementations that libraries do not change it. Names are ususlly
used to identify certain portions of models and hook up the code properly.
Thats something that provide standard linking mechanisms between artists and
programmers works.
Please also note that we use cache becuse we otherwise were loading PageLOD
files twice. Is it reasonable ?. Even if we skip intersections, other
situations are also possible. For example we may have few highly detailed
special PageLOD tiles with ariports which we want to preload and keep in
memory for whole application runtime. So we modify readFileCallback to work
for such cache and return these preloaded models each time thery are
requested. Their names will be changed as well and "NeedToRemove" scheme
stops working anyway.
With all due Respect ;-)
Wojtek Lewandowski
----- Original Message -----
From: "Robert Osfield" <robert....@gmail.com>
To: "OpenSceneGraph Users" <osg-...@lists.openscenegraph.org>
Sent: Monday, November 23, 2009 2:59 PM
Subject: Re: [osg-users] Refactoring DatabasePager
NeedToRemovestringflagging technique
Robert Osfield
On Mon, Nov 23, 2009 at 2:34 PM, Wojciech Lewandowski
<lewan...@ai.com.pl> wrote:
> But I would strongly defend merits of arguments in the post. You say we did
> wrong, but whats you recommendation on mixing many intersections with
> rendering ? Its very common scenario. How should we tackle it, if current
> approach is wrong ?
The right approach is a difficult one. Getting a paged scene graph to
work with intersections at highest resolutions and at the same time
manage things for rendering with requires just the appropriate LOD
child for the needs of visuals is awkward. I know often vis-sim apps
don't even try to mix the two, and have a separate process entirely
for dealing intersections as for doing the visuals. Some sims even
run the visuals and intersection testing on entirely different
machines. Other sims use entirely separate databases for intersection
testing and visuals. Then there are others that use a height field
for height above terrain testing...
If you do want to mix the two then you need to ask questions about how
you want to do intersections and how your thread them. Do you do
intersections in the main loop? In a separate thread? Do you run the
intersections multi-threaded? I.e. multiple intersections traversals
at one time? Should the intersection traversal wait to load external
tiles to get the result, or should then just return the intersections
for what is already loaded into memory.
What approach you will want to use will depend upon all of this.
> Second important issue for me is usage of _name in scene graph. I always
> expected that _name is reserved for users and its a normal rule in all Scene
> Graph implementations that libraries do not change it. Names are ususlly
> used to identify certain portions of models and hook up the code properly.
> Thats something that provide standard linking mechanisms between artists and
> programmers works.
I agree, but... in this instance the DatabasePager's algorithm was
about deleting a subgraph that would no longer have any role to play
in the applications life so the changing of name should never have got
outside that algorithm as the subgraph would be just deleted. So it
is in theory just a black box, how it does it's job shouldn't effect
anything else. Alas in this case it looks like the algorithm in
DatabasePager is flawed.
> Please also note that we use cache becuse we otherwise were loading PageLOD
> files twice. Is it reasonable ?.
Using a cache to prevent a subgraph from being deleted defeats the
load balancing that the DatabasePager will be attempting to do, so
it's a dangerous thing to do - it's a recipe for relentless growth in
memory usage.
> Even if we skip intersections, other
> situations are also possible. For example we may have few highly detailed
> special PageLOD tiles with ariports which we want to preload and keep in
> memory for whole application runtime. So we modify readFileCallback to work
> for such cache and return these preloaded models each time thery are
> requested.
Use of the cache in conjunction with a paged database should be used
very sparingly and for only very specific types of assets. It does
also open the question of how DatabasePager should deal with such
datasets, without lots of reflection on the issue I can't say. I can
say in the design and development of DatabasePager I have made the
assumption that it'd be the master of the PagedLOD's and manage all
reading and expiring, and not have code on the outside managing things
in a parallel.
It's worth noting that PagedLOD has settings that allow you to control
what happens with expiry - so you can individually switch off the
expiry.
> Their names will be changed as well and "NeedToRemove" scheme
> stops working anyway.
I suspect the "NeedToRemove" scheme is only scratching the surface of
the issue of apps that try to mix paging and caching.
Glenn Waldron
Hi Wojtek,
On Mon, Nov 23, 2009 at 2:34 PM, Wojciech Lewandowski
<lewan...@ai.com.pl> wrote:
> But I would strongly defend merits of arguments in the post. You say we didThe right approach is a difficult one. Getting a paged scene graph to
> wrong, but whats you recommendation on mixing many intersections with
> rendering ? Its very common scenario. How should we tackle it, if current
> approach is wrong ?
work with intersections at highest resolutions and at the same time
manage things for rendering with requires just the appropriate LOD
child for the needs of visuals is awkward. I know often vis-sim apps
don't even try to mix the two, and have a separate process entirely
for dealing intersections as for doing the visuals. Some sims even
run the visuals and intersection testing on entirely different
machines. Other sims use entirely separate databases for intersection
testing and visuals. Then there are others that use a height field
for height above terrain testing...
Wojtek,
The "keep it separate" approach is what we use in osgEarth. The idea is to fetch terrain tiles directly, based on your target sampling resolution, instead of traversing the whole LOD hierarchy. Take a took at the ElevationManager utility. Perhaps it can provide some inspiration:
http://wush.net/trac/osgearth/browser/trunk/src/osgEarthUtil/ElevationManager
Glenn
Chris 'Xenon' Hanson
>> Second important issue for me is usage of _name in scene graph. I always
>> expected that _name is reserved for users and its a normal rule in all Scene
>> Graph implementations that libraries do not change it. Names are ususlly
>> used to identify certain portions of models and hook up the code properly.
>> Thats something that provide standard linking mechanisms between artists and
>> programmers works.
> I agree, but... in this instance the DatabasePager's algorithm was
> about deleting a subgraph that would no longer have any role to play
> in the applications life so the changing of name should never have got
> outside that algorithm as the subgraph would be just deleted. So it
> is in theory just a black box, how it does it's job shouldn't effect
> anything else. Alas in this case it looks like the algorithm in
> DatabasePager is flawed.
This is the part that I couldn't tell from a simple reading of the code. Once a node is
renamed to "NeedToRemove", under normal circumstances (ignoring the cache trick above), is
there any possibility that that node can "return to life"? It sounds like you're saying
there is no possibility, in which case I sort of agree with Robert that altering the name
is fair game, as the node is now "the walking dead" and anything is permissible.
--
Chris 'Xenon' Hanson, omo sanza lettere Xenon AlphaPixel.com
PixelSense Landsat processing now available! http://www.alphapixel.com/demos/
"There is no Truth. There is only Perception. To Perceive is to Exist." - Xen
Wojciech Lewandowski
>> How should we tackle it, if current
>> approach is wrong ?
[..]
> If you do want to mix the two then you need to ask questions about how
> you want to do intersections and how your thread them. Do you do
> intersections in the main loop? In a separate thread? Do you run the
> intersections multi-threaded? I.e. multiple intersections traversals
> at one time? Should the intersection traversal wait to load external
> tiles to get the result, or should then just return the intersections
> for what is already loaded into memory.
> What approach you will want to use will depend upon all of this.
Our case is following: We mostly have to update camera position and some
objects around camera. And ocassionaly we reposition a camera to new
location in the world. We are doing our intersections in update traversal. I
assume its allowed to use IntersectVisitor in update and load new tiles
then. We load many objects in update traversal.
And believe it or not but our scheme was working. Cache kept nodes bit
longer than DatabasePager wanted, but not used tiles were eventually freed.
I know it did work, because we made quick dirty fix by renaming
"NeedToRemove" nodes to empty string name when fetching PagedLODs back to
activePageLOD list from cache (described in Pawel Ksiezopolski post I
mentioned earlier). Unfortuantely this fix was not appropriate as elegant
submission. Sharing loaded PageLOD tiles through cache was actually working
as some preload for DatabasePager. DatabasePager still could load whatever
it wanted.
>> Second important issue for me is usage of _name in scene graph. I always
>> expected that _name is reserved for users and its a normal rule in all
>> Scene
>> Graph implementations that libraries do not change it. Names are ususlly
>> used to identify certain portions of models and hook up the code
>> properly.
>> Thats something that provide standard linking mechanisms between artists
>> and
>> programmers works.
> I agree, but... in this instance the DatabasePager's algorithm was
> about deleting a subgraph that would no longer have any role to play
> in the applications life so the changing of name should never have got
> outside that algorithm as the subgraph would be just deleted. So it
> is in theory just a black box, how it does it's job shouldn't effect
> anything else. Alas in this case it looks like the algorithm in
> DatabasePager is flawed.
>> Please also note that we use cache becuse we otherwise were loading
>> PageLOD
>> files twice. Is it reasonable ?.
> Using a cache to prevent a subgraph from being deleted defeats the
> load balancing that the DatabasePager will be attempting to do, so
> it's a dangerous thing to do - it's a recipe for relentless growth in
> memory usage.
Well yes, but in our case PagedLOD usage timeframe should be extended to
period when the tiles were used for intersections and cache maybe in bit
hacky way provided this prolonged life time. And use of cache is not direct
cause of leaks becaus osgDB cache is freed when node ref_count reaches zero
so when DatabasePager & IntersectionVisitor stopped using the tile it was
effectively removed from the cache as well.
Memory leaks were the result of extra NeedToRemove nodes that appeared in
activePageLOD lists when some nodes were resurected from Cache because
camera moved over them again. I know its sophistry, but in some way, it was
more effective than native DatabasePager management because it loaded nodes
in no time while DatabasePager would have to load them from disk in this
case ;-).
>> Even if we skip intersections, other
>> situations are also possible. For example we may have few highly detailed
>> special PageLOD tiles with ariports which we want to preload and keep in
>> memory for whole application runtime. So we modify readFileCallback to
>> work
>> for such cache and return these preloaded models each time thery are
>> requested.
> Use of the cache in conjunction with a paged database should be used
> very sparingly and for only very specific types of assets. It does
> also open the question of how DatabasePager should deal with such
> datasets, without lots of reflection on the issue I can't say. I can
> say in the design and development of DatabasePager I have made the
> assumption that it'd be the master of the PagedLOD's and manage all
> reading and expiring, and not have code on the outside managing things
> in a parallel.
Well I thought that if PagedLOD class is public and offers public interface
and methods we could use it as long as we do not hit compiler errors.
Perhaps different set of classes should be created for use in more VPB
specific closed way.
> It's worth noting that PagedLOD has settings that allow you to control
> what happens with expiry - so you can individually switch off the
> expiry.
I have not thought about it. I will have to learn what we can do using this
mechanism, perhaps we could do something smarter.
Wojtek
Wojciech Lewandowski
I disagree, when Scene is available in update traversal for various visitors
you could expect that some user may want to do some specific decoration of
the PageLOD node. Modify texture attributes, attach some objects etc. So
once you allowed user to get to PagedLOD via update traversal vistiors you
should expect that this PagedLOD ref_count increases. In such case
DatabasePager should not longer assume it posses this node completely.
Other case could be following. We overrite ReadFileCallback and generate and
reuse some procedural generic tile (for example spherical sector of ocean
mesh). This is again a case where PageLOD is not directly loaded by
DatabasePager but only used by it. So DatabasePager should behave politely
with this node...
Cheers,
Wojtek
Wojciech Lewandowski
----- Original Message -----From: Glenn WaldronSent: Monday, November 23, 2009 5:41 PMSubject: Re: [osg-users] Refactoring DatabasePagerNeedToRemovestringflagging technique
Robert Osfield
On Mon, Nov 23, 2009 at 5:22 PM, Wojciech Lewandowski
<lewan...@ai.com.pl> wrote:
> Our case is following: We mostly have to update camera position and some
> objects around camera. And ocassionaly we reposition a camera to new
> location in the world. We are doing our intersections in update traversal. I
> assume its allowed to use IntersectVisitor in update and load new tiles
> then. We load many objects in update traversal.
The crucial part you don't talk about is the performance constraints.
Can you pause a frame for hundreds of milliseseconds while the
required high res tiles are pulled in off disk? Or are you trying to
stick with a solid 60Hz? Loading any tiles in the update phase is not
possible if you want to keep to a solid 60Hz.
> And believe it or not but our scheme was working. Cache kept nodes bit
> longer than DatabasePager wanted, but not used tiles were eventually freed.
> I know it did work, because we made quick dirty fix by renaming
> "NeedToRemove" nodes to empty string name when fetching PagedLODs back to
> activePageLOD list from cache (described in Pawel Ksiezopolski post I
> mentioned earlier). Unfortuantely this fix was not appropriate as elegant
> submission. Sharing loaded PageLOD tiles through cache was actually working
> as some preload for DatabasePager. DatabasePager still could load whatever
> it wanted.
I think something that might provide a "work around" such as replacing
the"NeedToRemove" is just a work around it's very unlikely to be a
solution of trying to use the DatabasePager beyond it's current design
assumptions.
>> Using a cache to prevent a subgraph from being deleted defeats the
>> load balancing that the DatabasePager will be attempting to do, so
>> it's a dangerous thing to do - it's a recipe for relentless growth in
>> memory usage.
>
> Well yes, but in our case PagedLOD usage timeframe should be extended to
> period when the tiles were used for intersections and cache maybe in bit
> hacky way provided this prolonged life time. And use of cache is not direct
> cause of leaks becaus osgDB cache is freed when node ref_count reaches zero
> so when DatabasePager & IntersectionVisitor stopped using the tile it was
> effectively removed from the cache as well.
> Memory leaks were the result of extra NeedToRemove nodes that appeared in
> activePageLOD lists when some nodes were resurected from Cache because
> camera moved over them again. I know its sophistry, but in some way, it was
> more effective than native DatabasePager management because it loaded nodes
> in no time while DatabasePager would have to load them from disk in this
> case ;-).
I can't understand why you want to do what you are trying to do, but
it still doesn't change the fact that DatabasePager hasn't been
designed with this in mind. We can certainly look to take this into
account in another rev of DatabasePager.
As a general note a good OS will actually cache file accesses,
especially if you don't overload virtual memory/swap space too much so
the cost of reloading a tile might not be quite as expensive as you
think... I know that when I benchmark under Linux I have the file
cache makes a huge difference, so I can to be careful about running
test apps in sequence.
> Well I thought that if PagedLOD class is public and offers public interface
> and methods we could use it as long as we do not hit compiler errors.
> Perhaps different set of classes should be created for use in more VPB
> specific closed way.
I don't think this is a VPB database specific issue. The problem you
are trying to solve is really to do with trying to juggle user manage
caching with how the DatabasePager manages expiry.
When I get on to reviewing the multiple viewpoint issue with
DatabasePager I'll have a think about the consequences of users
caching subgraphs
Robert.
Robert.
Wojciech Lewandowski
> The crucial part you don't talk about is the performance constraints.
> Can you pause a frame for hundreds of milliseseconds while the
> required high res tiles are pulled in off disk? Or are you trying to
> stick with a solid 60Hz? Loading any tiles in the update phase is not
> possible if you want to keep to a solid 60Hz.
You are right, we do not keep 60 hz then. But its rare case, we prepared our
ReadCacheCalback to wait for highest level of detail only when camera is
repositioned by more than some margin distance. There is an assumption that
warp move will not keep decent LOD anyway, so then we trade hz for qualtiy,
and force HIGHEST_LEVEL_OF_DETAIL to properly compute initial elevations and
preload highest level of detail for camera neighbourhood. Once the camera is
moved and new initial conditions are met we switch back to highest
avaialable PageLOD because we are sure that preloaded tiles will be fairly
high level and Paging will offer fairly good continuous intersection results
that will fit into our wiggle room. This is actualy a hybrid approach but I
have not mentioned it because it would complicate things even more.
> I think something that might provide a "work around" such as replacing
> the"NeedToRemove" is just a work around it's very unlikely to be a
> solution of trying to use the DatabasePager beyond it's current design
> assumptions.
I agree, but such replacement of original name seems like similar trick to
me, thats why I proposed to get rid of the cause completely. This is what
this thread is about, isn't it ?
>> Memory leaks were the result of extra NeedToRemove nodes that appeared in
>> activePageLOD lists when some nodes were resurected from Cache because
>> camera moved over them again. I know its sophistry, but in some way, it
>> was
>> more effective than native DatabasePager management because it loaded
>> nodes
>> in no time while DatabasePager would have to load them from disk in this
>> case ;-).
> I can't understand why you want to do what you are trying to do, but
> it still doesn't change the fact that DatabasePager hasn't been
> designed with this in mind. We can certainly look to take this into
> account in another rev of DatabasePager.
Thank You.
> As a general note a good OS will actually cache file accesses,
> especially if you don't overload virtual memory/swap space too much so
> the cost of reloading a tile might not be quite as expensive as you
> think... I know that when I benchmark under Linux I have the file
> cache makes a huge difference, so I can to be careful about running
> test apps in sequence.
True. On the other hand, with large datasets like Paged 400 GB database
constantly moving back and forth this genaral cache may have much worse hit
ratio then application cache tuned for the particular usage patterns. With
OS cache the terrain LOD switching was still more distracting than when we
turned on both OSG caches.
>> Well I thought that if PagedLOD class is public and offers public
>> interface
>> and methods we could use it as long as we do not hit compiler errors.
>> Perhaps different set of classes should be created for use in more VPB
>> specific closed way.
> I don't think this is a VPB database specific issue. The problem you
> are trying to solve is really to do with trying to juggle user manage
> caching with how the DatabasePager manages expiry.
Maybe I am. But additionally I also want to stress the fact: that if some
node is no longer present in current viewer scenegraph, it does not exactly
mean its no longer used. And assumption that DatabasePager can do anything
with such node, is at least a dangerous practice to me.
> When I get on to reviewing the multiple viewpoint issue with
> DatabasePager I'll have a think about the consequences of users
> caching subgraphs.
Thank You. Does it mean I should try to prepare the proposed submission or
not ?
Wojtek
Robert Osfield
On Tue, Nov 24, 2009 at 1:33 PM, Wojciech Lewandowski
<lewan...@ai.com.pl> wrote:
>> When I get on to reviewing the multiple viewpoint issue with
>> DatabasePager I'll have a think about the consequences of users
>> caching subgraphs.
>
> Thank You. Does it mean I should try to prepare the proposed submission or
> not ?
You can submit changes, I can always just use these as another point
of information when doing my investigation even if the code doesn't
make it into svn/trunk.
Robert.
Dan Small
I'd like to jump in on this conversation. We are currently in the camp of doing intersection testing against a non-rendered, high-resolution, non-paged terrain model, and then using a PagedLOD terrain model for viz. From all the comments, it seems like trying to combine the IntersectionVisitor and the database pager is a bad idea if you are also wanting to visualize the same scene graph. Our application is similar to Wojciech's, in that we don't want to pay the overhead of reloading the highest level LOD every time we do an intersection test. We are extracting regular gridded height fields off the terrain data, and then doing line lots of sight checks between the height field points and some other point in the scene.
I have two questions relevant to this thread:
1) Does the task of combining the database pager with the IntersectionVisitor (while ideally caching the highest LOD tile) get easier if you're not actually rendering the scenegraph?
2) Again with respect to a non-rendered, PagedLOD terrain, do you see any problems with running multiple threads where IntersectionVisitor is executed in those threads? We currently do this against our own static BSP tree representation of the scenegraph, and it works fine.
We would very much like to do these analyses on very large terrains, but even at 64 bits we will very easily run out of RAM when we try to load them. This makes using an OSGDEM/PagedLOD terrain for the intersection testing the next logical step.
Thanks,
Dan
Robert Osfield
If you were having issues related to the issues discussed in this thread please check out the svn/trunk and see if the issues are now addressed.
Thanks,
Robert.
Wojciech Lewandowski
----- Original Message -----From: Robert Osfield
Wojciech Lewandowski
Robert Osfield
Hi Robert,Our testing shows that everything looks good now. We will be doing some more work on the sim next week and do more testing on this occasion. We will let you know in case something suspicious pop up. But lets keep finger crossed it won't.
Good to hear things are now working. Thanks for the testing.
Robert.