vsg::read(..), vsg::External, vsg::ObjectCache and vsg::Operation/OperationThreads

[Robert Osfield]

I have merged my work on the External branch to VSG master, some of the new features introduced mirror ones in the OSG, but also ones that are quite novel, these new features are all connected so I'm introducing in this thread.  The work is also part of the wider database paging work that is still underway. I'm trying out googlegroups ability to provide formatting a links, so click on the highlighted text for links into the relevant VSG code.

1) vsg::read()

The first element to introduce is a new convenience functions are found in include/vsg/io/read.h

       // load a single file

       ref_ptr<Object> vsg::read(Path& filename, ref_ptr<const Options> options = {});

       // load multiple files, possible in parallel and return a map of <filename, object>

       vsg::PathObjects vsg::read(Paths& filenames. ref_ptr<const Options> options = {});

For convenience there is also a casting version of the single file read, to use it you do:

       auto model = vsg::read_cast<vsg::Node>("mymodel.vsgt");   // auto will map to a ref_ptr<vsg::Node>

     

If the root object loaded from a file isn't a vsg::Node, or subclass from vsg::Node then a null ref_ptr<> will be returned and the loaded object(s) automatically deleted.

2) vsg::OperationThreads

The multi-file version of vsg::read() reads single threaded in the calling thread if a null options parameter is passed in (default) or when the vsg::Options object doesn't have a vsg::OperationThreads object assigned to it.  To run the multi-file version if a parallel you use:

       auto options = vsg::Options::create();

       options->operationThreads = vsg::OperationThreads::create(8); // create 8 threads to run vsg::Operations with such as ReadOperations used by read().

       auto pathObjects = vsg::read(filenames, options); // read filenames

       for(auto& [filename, object] : pathObjects)

       {

              /// do what I want with the objects load, note object could be null if the file can't be read.

       }

vsg::Options and vsg::OperationThreads are both ref counted, and the create() return a ref_ptr<> so when the options goes out of scope and ref count goes to 0 the Options and in turn OperationThreads will be deleted, and with it all the threads will told to cooperative finish and std::thread::join() the thread doing the destruction.

3) vsg::ObjectCache

Similar in concept to the osgDB::ObjectCache the vsg::OjbectCache class essentially manages a thread safe map between filename as objects loaded from these files.  To enable the use of the ObjectCache you assign one to the Options object you pass into your read call i.e.

    auto options = vsg::Options::create();

    options->objectCache = vsg::ObjectCache::create(); // assign an ObjectCache to options

    auto node = vsg::read_cast<vsg::Node>("mymodel.vsgt", options);

The vsg::read* will automatically check whether the "mymodel.vsgt" is already in the cache, if it has it'll return object returned the cached object.  If the object isn't there it'll automatically insert it, so that any subsequent calls can then reuse it.

The ObjectCache is ref counted for you so again you won't need to worry about lifetime, once the Options goes out of scope and is deleted the ObjectCache will be deleted as well if it's not referenced anywhere else.  For instance if you do:

      ref_ptr<Object> object;

      { // create local scope

            auto options = vsg::Options::create(); // optoins will be of type ref_ptr<vsg::Options>

            options->objectCache = vsg::ObjectCache::create(); // ObjectCache created on heap and ref counted by Options

            object = vsg::read("myfile.vsgb")

      } // options goes out of scope and destructs the Options object, then ObjectCache gets unref'd and gets destructed, all references held in the internal map will get unref'd and cleaned up

      // here the any Object loaded in the above local scope will still be assigned to the object variable, but all other references to it have been removed

If you want your ObjectCache to be used over multiple read calls you reuse the same Options object, or assign the same ObjectCache to multiple Options objects.

4) vsg::External

The vsg::External is something that is doesn't have any direct equivalent in the OSG and unlike the classes it's actually part of the scene graph rather than a utility for loading/process one.  The role of vsg::External is provide a mechanism for storing scene graph objects in separate files on disk, and when used in conjunction with the vsg::ObjectCache the ability to share objects between separate scene graphs loaded from separate files. 

A single vsg::External object have have multiple file references, and when a vsg::OperationThreads object is assigned to the vsg::Options used when reading a scene graph with an vsg::External in it, the reading of the external files will be done multi-threaded.  Both the ObjectCache and OperationThreads can be used together just by assigning both the Options object that you pass into the vsg::read call.

The serializer for vsg::External provides support for reading/writing of the filenames and managing the reading and writing of the files for you, and will automatically manage sharing of objects in the rest of the serialized scene graph that contains the External.  To enable this what you do is put the objects you want to shared in a vsg::Objects and assign it to vsg::External, then attach this External object to the root of graph that you want to serialize.

An example of the way you set this sharing up can be found in the External branch of osg2vsg/applications/osg2vsg/osg2vsg.cpp. 

The above example is more of hack to test the feature rather than a full blown example of how I expect this feature to be deployed.  Where I expect this to be used will be:

• Storing pipeline related state, such as vsg::GraphicsPipeline, vsg::PipelineLayout, vsg::Shader etc.
• Storing texture data, such as tree or road textures that get reused many times in different models or tiles in paged database
• Storing geometry data, such as common geometry elements like fences, lampposts, cars, people, trees that you want to reuse across different models/tiles.
• Whole subgraphs where you want to split a large database, such as a city, into more management chunks to fit within file system constraints (or even storing files in a github rep :-)
  
• Enable scene graphs to be broken up to optimize the ability to multi-thread reading, something that could particularly important for database loaded across http.

--

I hope this makes enough sense of where I'm going with this functionality.  It's still really early days for this functionality so there me uses or issues that I haven't thought of or haven't coded for yet.  As I get further along the database paging work I should get the chance to test more of this functionality out, this should help shake down problems as well give others a bit more guidance on how these features are used.