"Nested" Sessions/Child Object Issues

[Russ McClelland]

I have a aggregation pattern that I am saving in RavenDB. An object can contain multiple children of the same type. These children can be shared across multiple parents. I've written a custom converter that breaks the graph into separate nodes and stores all the objects as individual documents. The parent's collection is converted to an array of "referenceID" : [some child ID].

When I read the child by itself, no problems because his "child" collection is empty. The exception it throws is: 

System.InvalidOperationException: Could not convert document fea3fb67-a601-4838-9659-bbcbdd63b88d to entity of type MyEntityType --->

 

System.InvalidOperationException: Could not convert document dd6c6fb7-af73-4d01-862a-91d3061daad9 to entity of type MyEntityType  --->

 

System.InvalidOperationException: Cannot add a data setter because on is already added

When I read the parent, it throws an exception when I get the child ID and make a call to go get the document with that ID. I wouldn't think there would be any issues because each call  opens and closes a new session. For my test example, I only have 2 children, so a total of three sessions (should) be opened.

        public override object ReadJson( JsonReader stream, Type objectType, object existingValue, JsonSerializer serializer )

        {

            JObject streamElements = JObject.Load( stream );

            Entity parent = new Entity( new Guid( streamElements["ObjectID"].Value<string>() ) );

            EntityBroker broker = this.GetBroker();

            

            foreach( JProperty referenceID in streamElements["ChildEntityIDs"].Values() )

            {

                parent.AddChild( broker.FindEntityWith( new Guid( referenceID.Value.Value<string>() ) ) );

            }

            return parent;

        }

This is the FindEntityWith method which is called for both the parent and the children:

            using( var session = this.Connection.OpenSession() )

            {

                orgs = session

                            .Query<Organization, EntitiesByOIDAndPerspectiveIndex>()

                            .Where( o => o.OID == objectID && o.Perspective == perspective )

                            .ToList();

            }

Any idea what causes this error? I only found one mention of it on the web, and it remained unanswered.

[Oren Eini (Ayende Rahien)]

What do your documents look like?

And what is the full exceptions that are being thrown?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Russ McClelland]

I attached a sample unit test project which replicates the problem. If I read a child with the custom converter, it works fine. When I read a parent, it reads the properties fine and creates the object. I then "recursively" make a call to read each child based on an ID. That fails with this exception tree:

{"Could not convert document units/291 to entity of type CustomConverters.Unit"}

   Inner Exception: {"Cannot add a data setter because on is already added"}

This is a sample child object:

{

    "$type": "CustomConverters.Unit,  CustomConverters",

    "Name": "Test Child 1",

    "PrimUnits": [],

    "OID": "b88fa84f-6eb6-4ba4-8532-26dec449a35a"

}

This is a sample parent:

{

    "$type": "CustomConverters.Unit,  CustomConverters",

    "Name": "Test Parent",

    "PrimUnits": [

        {

            "referenceID": "b88fa84f-6eb6-4ba4-8532-26dec449a35a"

        },

        {

            "referenceID": "b712935a-e33c-4a06-b23c-6fb2de923378"

        },

        {

            "referenceID": "f010582e-80bb-4465-99d6-1260d4c02ccc"

        }

    ],

    "OID": "bf569808-32bd-452c-8cd4-f75bf5c1ae2e"

}

[Oren Eini (Ayende Rahien)]

Well, to start with, you really shouldn't do this sort of things.

      public override void WriteJson( JsonWriter stream, object value, JsonSerializer serializer )

        {

            Unit org = value as Unit;

            UnitBroker broker = this.GetBroker();

            org.Units.ForEach( subunit => broker.SaveUnit( subunit ) );

Here you are calling on the session, during the SaveChanges process and create a new session, and save some stuff.

This will break the transaction, it mess up the behavior of the client and can cause surprising behavior.

Your read behavior is actually worse, you are doing a query, which recursively does more queries, etc.

The problem with the error is that you are executing things recursively, and that isn't supported, so it throws.

I would recommend seriously rethinking the data model, it doesn't make sense, and it doesn't make use of any of the RavenDB features that are meant to deal with this kind of feature explicitly.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Russ McClelland]

What are the features meant to handle this sort of requirement (objects that need to be referenced in multiple root graphs)? I would have thought this was safe because each trip through the loop, we create a new session. Semantically, this shouldn't be any different than a second client creating a session and reading documents.

As with the read, is there a better way in RavenDB to load the "child" documents when I don't know what their IDs are until I start reading the parent?

[Ryan Heath]

You could use includes to load objects referenced by other objects. 

 

See "one to many includes" on https://ravendb.net/docs/article-page/3.5/csharp/indexes/querying/handling-document-relationships

// Ryan

On Tue, 4 Jul 2017 at 15:41, Russ McClelland <russ.mc...@gmail.com> wrote:

What are the features meant to handle this sort of requirement (objects that need to be referenced in multiple root graphs)? I would have thought this was safe because each trip through the loop, we create a new session. Semantically, this shouldn't be any different than a second client creating a session and reading documents.

As with the read, is there a better way in RavenDB to load the "child" documents when I don't know what their IDs are until I start reading the parent?

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Russ McClelland]

Looking at these examples, it looks like I would need a second collection on my parent object just to store only the IDs of the objects in the real collection. I don't have that (nor do I have the code to control adding that). Would this be easier without using the Session? Seems like this is the object that doesn't like having multiple requests happening simultaneously.

[Russ McClelland]

The other thing I don't understand about the session, is why opening separate sessions breaks the original session. The code example intentionally opens a new session for each child. While I realize this seems inefficient (and may be ridiculous for RavenDB), but I'm not understanding why we can't have multiple open sessions on the same database without causing read issues. I'm new to Raven, so I must be missing something...

[Russ McClelland]

"Here you are calling on the session, during the SaveChanges process and create a new session, and save some stuff.

This will break the transaction, it mess up the behavior of the client and can cause surprising behavior."

How is Raven normally used with multiple client threads each creating a session in something like a standard web service? In this case, you don't want each thread waiting for other threads to finish. Each thread may be calling the same or different methods on the service that are all accessing the same database.

[Oren Eini (Ayende Rahien)]

Because there is a thread static value that is used there for interaction between different pieces of the code, and your recursive call is messing it up.

There is no expectation that any code will run recursively during session operations.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Tue, Jul 4, 2017 at 9:11 PM, Russ McClelland <russ.mc...@gmail.com> wrote:

The other thing I don't understand about the session, is why opening separate sessions breaks the original session. The code example intentionally opens a new session for each child. While I realize this seems inefficient (and may be ridiculous for RavenDB), but I'm not understanding why we can't have multiple open sessions on the same database without causing read issues. I'm new to Raven, so I must be missing something...

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

There is no problem with concurrency, there is a problem when you are in the middle of operation and suddenly start another operation.

The code is no re-enterant safe, nor is it meant to be

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Russ McClelland]

But I'm using a completely different Session object. The operations are being called on 2 different instances attached to the same DocumentStore. Are sessions thread safe? Are they really not different instances? It looks like they have different IDs. (e.g. {8dc3cf6c-e8a2-462b-8a01-1e7169db9afb} and {86a9102f-9a92-49e2-a3bb-b252f43d6e8d}). I've been thinking of them as being analogous to "connections" in a SQL Server context.

[Russ McClelland]

That makes sense. What happens when 2 completely different clients call the same method at the same time, again using different sessions? Does that not cause the same issues? Thinking about a SQL Server analog, I could call the same method hundreds of times simultaneously and because each call opens a connection (or gets one from the pool), they are executing completely independent of each other. How do you do something similar in Raven?

[Ryan Heath]

> What happens when 2 completely different clients call the same method at the same time, again using different sessions?

That would be no problem, since their operations would run on different threads.

Having multiple sessions alive on the same thread is not the issue at hand.

In your case, *while* you are reading an (json) object you are re-entering the read json methods when reading the child objects.

Re-entering of reading json objects is not supported.

You could create a new thread when reading the child objects but that would be highly inefficient.

A better way would be to have a reference id list and an object list property.

Your unitbroker could populate the object list.

 

// Ryan

--

[Russ McClelland]

Thanks for trying to shed some light on this, I really appreciate the feedback.

I would think that since I have different sessions, they would each get a different converter, and each session reading would get its own JsonReader because each session would have its own stream of data to read. Is something assigning the same instance of JsonReader to every process on the same thread? Otherwise, I don't see how the example would be re-entrant.

// Ryan

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Russ McClelland]

"A better way would be to have a reference id list and an object list property. Your unitbroker could populate the object list.

I don't have the ability to add extra properties. Seems like having to keep 2 lists in sync with each other isn't a very desirable pattern. Any logic in the AddUnit() method would need to be duplicated in the AddUnitID(), and in fact not applicable. Outside the context of deserializing from the database, adding a Guid to the list of keys is not a valid operation because the ID would be meaningless.  Likewise, having the AddUnitID() actually do something would require a dependency on the persistence frameworks in our otherwise unencumbered libraries...

[Oren Eini (Ayende Rahien)]

The problem is on the _client side_. A thread local variable is used to hold some data for the duration of the deserialization of an entity.

It is not expected to run recursively.

You can disable that by setting Conventions.PreserveDocumentPropertiesNotFoundOnModel = false;

That is not recommended and I would strongly urge you to use a different model. You are going to be hitting the database with a LOT of queries, in the most sub-optimal manner possible.

--

[Oren Eini (Ayende Rahien)]

The actual shared instance here is the contract resolver. Because this can be called concurrently, it needs some way to stash data for the current operation, this is done here:

https://github.com/ravendb/ravendb/blob/f3b5f3a186d07776bf38bf9effab4d7d75d5c647/Raven.Client.Lightweight/Document/DefaultRavenContractResolver.cs#L32

You are then trying to do this recursively, and this is detected and fail. 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ryan Heath]

Maybe you could use an intermediate object in your unit broker. 

This object should be easily cloned from and to the actual unit object. 

And only be used in your unit broker object. 

In a sense you are already doing that with the custom converter. But having a concrete cloneable object would make things a lot easier: 

no need to change ravendb default settings, no unneeded for multiple round trips to the db, separation of persistence model and actual model that allows you to use a string Id property instead of an OID Guid property ...

There are of course downsides of this approach, that may or may not apply for your application. 

From the outside it looks like you are imposing an relational architecture (the actual objects) on the document oriented philosophy of ravendb (and document dbs in general). This might bite you sooner or later. 

// Ryan

On Wed, 5 Jul 2017 at 01:52, Russ McClelland <russ.mc...@gmail.com> wrote:

"A better way would be to have a reference id list and an object list property. Your unitbroker could populate the object list.

I don't have the ability to add extra properties. Seems like having to keep 2 lists in sync with each other isn't a very desirable pattern. Any logic in the AddUnit() method would need to be duplicated in the AddUnitID(), and in fact not applicable. Outside the context of deserializing from the database, adding a Guid to the list of keys is not a valid operation because the ID would be meaningless.  Likewise, having the AddUnitID() actually do something would require a dependency on the persistence frameworks in our otherwise unencumbered libraries...

--

[Russ McClelland]

That's exactly what I was thinking as well after our last messages. I created a decorator that can be serialized in/out of Raven that deals in just IDs. I'm in the middle of testing is to make sure I can fault in multiple levels correctly.

I also started looking at Cosmos DB, which has support for object graphs, which in effect, is exactly what I'm trying to do.

[Russ McClelland]

I created the decorator and tried using the Query method on the session, but it isn't loading the included documents. Am I missing a setting or convention somewhere in order to do that?

                orgs = session

                            .Query<UnitDecorator>()

                            .Customize( u => u.Include<UnitDecorator>( su => su.UnitIDs ) )

                            .Where( u => u.UnitID == requestedUnitID )                            

                            .ToList();

I would have thought this would load the first level "children", which have also been converted to decorators in order to be saved. They are indeed saved, but not included in the session results.

I attached an updated version of the test project.

[Ryan Heath]

You probably need a list<string> instead of list<Guid> in the unitdecorator.

Also make the Id as a normal property, remove Guid UnitID from unitdecorator.

push the conversion from and to Guid into the cloning methods.

HTH 

// Ryan

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ryan Heath]

Change FindUnitWith into (note: gmail compiled!)

public Unit FindUnitWith( Guid requestedUnitID, string perspective = "" )

        {

            Unit newUnit = null;

            using( var session = this.Connection.OpenSession() )

            {

                var orgs = session
.Include<UnitDecorator>(ud => ud.UnitIDs)
.Load<UnitDecorator>(requestedUnitID.ToString());

                if( orgs == null )

                {

                    return null;

                }

                newUnit = orgs.AsUnit();

newUnit.Units = session.Load<UnitDecorator>(orgs.UnitIDs).Select(ud => ud.AsUnit()).ToList();

            }

            return newUnit;

        }

HTH

// Ryan

[Russ McClelland]

That did it! Any idea why the Customize() didn't work? I think I saw that in another example for customizing queries. I had original used that example because in our "real" code, we aren't selecting by IDs, but using several properties instead. 

[Ryan Heath]

I changed it into Load because you can only get at most one object when requesting by Id.

But the query will work too when Units is a list of string instead of list of Guid.

// Ryan

On Thu, Jul 6, 2017 at 12:20 AM, Russ McClelland <russ.mc...@gmail.com> wrote:

That did it! Any idea why the Customize() didn't work? I think I saw that in another example for customizing queries. I had original used that example because in our "real" code, we aren't selecting by IDs, but using several properties instead. 

--