Fast acces to ScalarTypes (leaf ) ?

[mr.j...@gmail.com]

Hi

I've been working with Platform,Foundation, Reflect,Persist for a while ( and I appreciate what you have done, great work !!! )

* Currently a pure binary serialization to a blob ( similar to ArchiveWriterMessagePack )
*  wxPropertyGrid integration 


My question is: what is the fastest way to access "ScalarTypes" or pretty much anything that can't be reached with "first level" static member pointer.

example :

ComicUniverse c;

1. serialize just m_HitPoints
c.m_Heroes[ 1 ].m_HitPoints

2. or serialize all in m_Heroes[1]
c.m_Heroes[ 1 ]


I'm currently thinking of a lookup table with keys built like this :

1: Crc32( ComicUniverse:heroes[1]:hp )  => c.m_Heroes[ 1 ].m_HitPoints

2: Crc32( ComicUniverse:heroes[1] )     =>   c.m_Heroes[ 1 ]

My understanding is I can't reach a "leaf pointer" in one go, I have to do a kind of traversal ( list of "node Pointers" ? ) is that correct ?
( In short what is the fastest way to reach a leaf from an Object pointer ? )

------- Network UDP

I want to build a highly optimized ( regarding bandwidth) publish/subscribe mechanism using Reflect and ENet ( http://enet.bespin.org/)

similar to what I want to build ( same same but different ;-)
http://savannah.nongnu.org/projects/certi
http://www.opendds.org/

In the same way you decoupled persist from reflect, I think a clear separation of Reflect, Serialization and network layer will give freedom to implement batch, compress, cache synchronization  AND special treatment for VIP objects =)

Any thoughts/suggestions on network-layer - UDP ?   ( My understanding is IPC is currently just IPCTCP ? )

/JB  

[Geoff Evans]

Hi JB,

So, are you working on these things?:

* Currently a pure binary serialization to a blob ( similar to ArchiveWriterMessagePack )
* wxPropertyGrid integration  

It wasn't clear from your original email what you were saying about those.

Regarding your other questions:

1)  Aggregated Field selection by syntax

This isn't supported at the moment, and its that way to try and save memory at runtime.  At the current moment we don't bake all aggregated metadata up into aggregating metadata.  By this I mean:

struct Foo {};

struct Bar { Foo f; };

struct Baz { Foo f; };

There is only one copy of the metadata for Foo.  To do what you propose, you would need to copy all of Foo's metadata up into Bar and Baz, and that would take more memory at runtime.

Devil's Advocate Question: What is the benefit of what you are proposing?  If you could write a syntax to select a specific field nested within one or more aggregates/arrays, why would you not just use C++ syntax to select that memory?  The driving thrust of Reflection/Introspection is so that you can write code that doesn't need to assume a particular field layout or aggregation.  Inventing a whole separate syntax to handle aggregates and array/associative-array item selection seems overcomplex (but this depends on the problem you are trying to solve).

2) Pubsub

I am all for keeping things modular, but I can't quite conceive of the design of what you want to build just from what you have included in this post.  Personally those simulation network libraries are overcomplicated.  Can you elaborate a bit more on how you want transmission/sync to work between objects in different address spaces?  (I assume this is the problem you are trying to solve).  I also find that approach interesting but I haven't settled in my head on a good approach to doing that with only reflection and reliable UDP.

-geoff

[mr.j...@gmail.com]

Hi,

 

So, are you working on these things?:

 

Yez ! under construction  (   blob, wxHeliumProperty )

https://lh3.googleusercontent.com/-7XGExC68pCA/VRMFcWyuzVI/AAAAAAAAAAc/ZxuQg3n1GVE/s1600/wxhelium.PNG

 

There is only one copy of the metadata for Foo.  To do what you propose, you would need to copy all of Foo's metadata up into Bar and Baz, and that would take more memory at runtime.

I know, but I'm extending the "register" information
 

Devil's Advocate Question: What is the benefit of what you are proposing?  If you could write a syntax to select a specific field nested within one or more aggregates/arrays, why would you not just use C++ syntax to select that memory?  The driving thrust of Reflection/Introspection is so that you can write code that doesn't need to assume a particular field layout or aggregation.  Inventing a whole separate syntax to handle aggregates and array/associative-array item selection seems overcomplex (but this depends on the problem you are trying to solve).

I want a simple method for describing changes to an object in the editor and a workhorse for serializing  just the difference....and I believe Reflect has almost everything I need to accomplish that.


I have implemented metalookup, an extension to register ( I'm not finished with dynamic arrays yet, changes can't be descried with just one crc32.. since items don't exist at compile time ;-)

int main()
{
    Initialize();
    Blob  blob;

    meta_lookup&  metalookup = meta_lookup::get_instance();

    ComicUniverse* pComic1= new ComicUniverse ();

    metalookup.add(  ComicUniverse::s_MetaClass );  // extension to register.....

    std::vector<uint32_t> vChgFields;  // crc to changed "field_paths"


// Kind of workflow
    if( editor )
    {
        // Editor  ( work with tmp_obj = pComic1->clone() )
        m_pWxHelium = new wxHeliumProperty(wxT("Helium"),wxPG_LABEL,  pComic1 );
        m_pWxHelium->get_changes( vChgFields );
    }
    else
    {
        // or hardcoded.. not a common use case

        pComic1->m_Heroes[ 3 ].m_Name = TXT("testing");
        pComic1->m_Heroes[ 3 ].m_HitPoints = 42;
        pComic1->m_Heroes[ 7 ].m_HitPoints = 24;

        vChgFields.push_back(Crc32("Heroes[3]:hp"));
        vChgFields.push_back(Crc32("Heroes[3]:name"));
        vChgFields.push_back(Crc32("Heroes[7]:hp"));
    }




 // Just, playing with Pointer &  Field operations...workhorse for blob:write and editor

    Pointer  ptr;
    const Field* field=0;
    metalookup.find( pComic1, vChgFields[0], ptr, field );
    uint32_t hp3 = ptr.As<uint32_t>();
    metalookup.find( pComic1, vChgFields[1], ptr, field );
    String name3 = ptr.As<String>();
    metalookup.find( pComic1, vChgFields[2], ptr, field );
    uint32_t hp7 = ptr.As<uint32_t>();

 

2) Pubsub

I am all for keeping things modular, but I can't quite conceive of the design of what you want to build just from what you have included in this post.  Personally those simulation network libraries are overcomplicated.  Can you elaborate a bit more on how you want transmission/sync to work between objects in different address spaces?  (I assume this is the problem you are trying to solve).  I also find that approach interesting but I haven't settled in my head on a good approach to doing that with only reflection and reliable UDP.

A simplified example of  Pub/Sub

// Write selected/changed fields to blob ( using metalookup.find + serialize_translator for each "field" )

     blob.write(pComic1,vChgFields);

 Publisher :

     ENet => send( blob ) => "subscribers"

 Subscriber:
     ENet=> receive( blob ) => find object =>  blob.read(object/changes); => display updates.....

 

This i just a brief description....In a few weeks (I hope) to have a small "Hello ComicUniverse" ready ( just a very basic test of building blocks Reflect blob, wxHeliumProperty, ENet pub/sub )

/JB

[Geoff Evans]

PropertyGrid integration would be cool :)  Keep in mind though that there is a reflection-driven properties system in Helium already: Inspect.  Inspect itself is toolkit agnostic, and the wxWidgets integration is currently implemented in the Editor project (all wx code is currently there).

Are you inventing your own binary blob format?  FWIW I really, really like msgpack and its somewhat widely supported.  The msgpack implementation that is in Persist right now is a work-in-progress.  Really the json support is the only fully functional Persist implementation.  I have been trying to think of a better way of abstracting different formats that how the Persist Archive classes are currently used.  Some way we could make Archive a template class instead of polymorphic would be nice.  That way one could just implement some template functions against the data format library (libbson, libmsgpack, rapidjson) intead of having to re-implement the core logic of traversing fields and data structures, which is tricky business!

Right now much of the code in Persist does conditionally serialize fields based on values in another instance of the same class.  For simple programs this ends up being the default instance, which is allocated at class registration and freed at unregistration.  aclysma and I have been working toward formalizing the use of the Object::GetTemplate() function for resolving the object appropriate for comparision during serialization.  This Template concept was added by the Lunar engine so this functionality within Reflect and Persist is pretty unstable/untested.  You might want to take a look and see if that might be less code to attain (a patch within Helium instead of a separate stack of code).

[Marc Hernandez]

On Fri, Mar 27, 2015 at 3:59 PM, Geoff Evans <gor...@gmail.com> wrote:

PropertyGrid integration would be cool :)  Keep in mind though that there is a reflection-driven properties system in Helium already: Inspect.  Inspect itself is toolkit agnostic, and the wxWidgets integration is currently implemented in the Editor project (all wx code is currently there).

Are you inventing your own binary blob format?  FWIW I really, really like msgpack and its somewhat widely supported.  The msgpack implementation that is in Persist right now is a work-in-progress.  Really the json support is the only fully functional Persist implementation.  I have been trying to think of a better way of abstracting different formats that how the Persist Archive classes are currently used.  Some way we could make Archive a template class instead of polymorphic would be nice.  That way one could just implement some template functions against the data format library (libbson, libmsgpack, rapidjson) intead of having to re-implement the core logic of traversing fields and data structures, which is tricky business!

  If it helps, Ive done exactly this and it works well, and you can use it for different things, not just archiving.  I used a macro + passing in a template thingy.  My next evolution of this is going to be adding attributes so you can tell the various walkers things about the data.  The type itself should probably return attributes too. 

  Something like (usually hidden behind macros):

template< typename TWALKER >

void walk( TWALKER *walk )
{

  WALK( m_health, AttDisplayMax( 1000.0f ) + AttNoPersist() );

}

  Leaf classes were a little weird.  I think I ended up having the root calling a virtual function walkLeaf, which had the job of then walking up the hierarchy. 

  Though, that said, Im now looking at clReflect which uses clang just parse the stuff, then you can do what you want with it. 

--

/// //

[Geoff Evans]

It looks like you are talking about how the templating works for harvesting the reflection data in the first place.  My point about templating the archive classes.  So I am talking about how one could separate the logic of walking the fields in type metadata from the mechanics of calling into the particular library that abstracts the data format.  Right now we have:

class Archive // abstract base class with some virtuals, shares code with child classes

class ArchiveJson // handles traversal mechanics and calls appropriate json library calls

class ArchiveBson // handles traversal mechanics and calls appropriate bson library calls

etc..

My point was: it would be nice if we could move the traversal portion of the child classes into the base Archive (or template), so that each child class is just implementing the calls to the specific library to read or write the appropriate data from the buffer/document.  I guess one way to do it would be to have an abstract class ArchiveFormat that could be subclassed for differente data formats... but I am not quite sure off the top of my head what the performance penalty for that would be.

The reason this is advantageous is that with the current design you have to essentially copy and paste the traversal logic when making a new data format, and if you fix a bug in one it doesn't carry over into the other formats.