Alternatives to transaction

[Steve Beaugé]

Hello,

I'm fully aware that inserting multiple document in a transaction is a
mongodb possibility (I understand the reasons behind). I'm however
looking for a simple solution to reach a basic degree of consistency.

Let me explain my scenario and my idea to compensate (I'm a new
mongodb user, apologize if I'm asking dumb questions) :

I'm building an application (C#) that works with an event sourcing
approach. Such applications are writing a lot of small "events" data
in a single collection :

{
EventID : "1234", <-- objectId
EventSourceID : "abc",
TimeStamp : "2011-xxxx"
Data : { val1: "vallue 1", val2 : "value 2" }
}

Most of time the application have to insert several of this event data
in one "transaction" (All have to succeeded or none).

As MOngodb has atomic operation on "one" document, I'm wondering if
trying to insert "one" composite object instead of a small object is a
good idea. In fact, instead of inserting "events", I'll insert
"commits" which have events as sub document :

{
commitId : "1234567890", <-- id
events: [{
EventID : "1234", <-- id
EventSourceID : "abc",
TimeStamp : "2011-xxxx"
Data : [{ val1: "vallue 1", val2 : "value 2"]
},
{
EventID : "4321", <-- id
EventSourceID : "abc",
TimeStamp : "2011-xxxx"
Data : [{ val1: "vallue 1", val2 : "value 2"]
},
{
EventID : "56789", <-- id
EventSourceID : "abc",
TimeStamp : "2011-xxxx"
Data : [{ val1: "vallue 1", val2 : "value 2"]
}]
}

Then, to retrieve set of elements, I can use standard $elemMatch
fonctions.

As I'm new to mongo, I have some questions about this approach :
1. Does it works ?
2. What is the impact in term of performance (after setting the
appropriate index of course)
3. Any better idea ?

[Sam Millman]

How big is data?

Technically it should work (embedded schema) but tbh your thinking too close to SQL with transactions, why are you trying to pump all *potentially* 8 into one query. Tbh I am not sure how much of a speed difference you will see between using on query or using 8.

Tbh there prolly wouldn't be much of a speed difference unless the querying is more natural when the events are embedded.

The upside to performance is that (in a distributed environment) the events given for a specific commit are all bunched together, not sure if that would help in your querying tbh.

As for better ideas, meh that is pretty much the two on market atm.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Steve Beaugé]

The data is not too big. It will be far more lower than the document
size limit per commit (My guess is that the largest commits will
hardly reach 1 MB).

I didn't precise in my first question, but my app will also be using
CQRS principles. That means I can accept small overhead when reading
and writing this events collection, as long as it stays under a
reasonable overhead. Up to 2 times longer then separate elements is
acceptable. I was actually fearing a major performance degradation (I
said I'm a newbie:p).

thx,
steve


PS: I'm not native english speaking, and I don't understand "tbh" and
"atm"... can you expand plz (or I should say please).

On 7 nov, 15:11, Sam Millman <sam.mill...@gmail.com> wrote:
> How big is data?
>
> Technically it should work (embedded schema) but tbh your thinking too
> close to SQL with transactions, why are you trying to pump all
> *potentially* 8 into one query. Tbh I am not sure how much of a speed
> difference you will see between using on query or using 8.
>
> Tbh there prolly wouldn't be much of a speed difference unless the querying
> is more natural when the events are embedded.
>
> The upside to performance is that (in a distributed environment) the events
> given for a specific commit are all bunched together, not sure if that
> would help in your querying tbh.
>
> As for better ideas, meh that is pretty much the two on market atm.
>

> > "mongodb-user" group.> To post to this group, send email tomongo...@googlegroups.com.

[Timothy Hawkins]

TBH= too be honest
ATM=at the moment

Sent from my iPad

> To post to this group, send email to mongod...@googlegroups.com.

[Sam Millman]

Oops sorry about the short hand. I can sometimes do it when I am feeling a little tired :).

The only bottleneck I can think of (if your not using straight to HDD writes) is on the network but since this data is going be really small it probably won't even make a peak on your data graphs.

To post to this group, send email to mongod...@googlegroups.com.

[Steve Beaugé]

Just have to implement my idea now ;)

Thanks Sam for sharing your thoughts, and thanks Timothy for expanding
the acronyms

On 7 nov, 15:33, Sam Millman <sam.mill...@gmail.com> wrote:
> Oops sorry about the short hand. I can sometimes do it when I am feeling a
> little tired :).
>
> The only bottleneck I can think of (if your not using straight to HDD
> writes) is on the network but since this data is going be really small it
> probably won't even make a peak on your data graphs.
>

[Steve Beaugé]

Hello,

I'm having a bit of troubles with querying my collection.

If I query using find(), I get (simplified):

{
"_id":{
"$uuid":"4e569437-227c-39a0-bf2b-8804949fd4bb"
},
"Sub":[
{
"SubId":{
"$uuid":"4fa8064f-349e-80b7-68de-af4c02c355be"
},
"RelatedId":{
"$uuid":"7bc745f9-77eb-4e7e-b3a3-1dfd0231545f"
}
}
]
}
{
"_id":{
"$uuid":"836B4352-B0D1-4615-98A0-79203712B571"
},
"Sub":[
{
"SubId":{
"$uuid":"21BB93D1-52B7-47B7-8E1A-73C9945D278A"
},
"RelatedId":{
"$uuid":"7bc745f9-77eb-4e7e-b3a3-1dfd0231545f"
}
}
]
}


now, I'm trying to get all "sub" item, having the same relatedid. How
can I do that ?

Thanks

> > > You received this message because you are subscribed to the Google Groups> > "mongodb-user" group.> To post to this group, send emailtomo...@googlegroups.com.> > To unsubscribe from this group, send email to>mongodb-user...@googlegroups.com.

[Sam Millman]

Is there a reason why you have $uuid field to the relatedid?

If not take it out and then you can do:

find({'Sub.RelatedId': "7bc745f9-77eb-4e7e-b3a3-1dfd0231545f"})

To post to this group, send email to mongod...@googlegroups.com.

[Sam Millman]

Oh hang on it is because you have a ObjectId there, should be fine to take that ObjectId out and just use string otherwise query using the object id construct as well.

[Steve Beaugé]

I'm writing data using the C# Driver. My source objects have property
of type "Guid", and I didn't handle the serialization process myself
(I can poste the code if it can help).

You suggest me to use find, but (not clear in my post I admit) my sub
is an array that can contains multiple elements, not all having the
same relatedId (but several can have the same relatedid).
Shouldn't the query use $elemMatch ?


I tried the following query :


db.Commits.find({ "$elemMatch" : { "Sub.RelatedId" :
"4fa8064f-349e-80b7-68de-af4c02c355be"}})

db.Commits.find({ "$elemMatch" : { "Sub.RelatedId" :
{"$uuid":"4fa8064f-349e-80b7-68de-af4c02c355be"}}})

but no one works... (no error, but no results)

On 8 nov, 10:46, Sam Millman <sam.mill...@gmail.com> wrote:
> Oh hang on it is because you have a ObjectId there, should be fine to take
> that ObjectId out and just use string otherwise query using the object id
> construct as well.
>

> On 8 November 2011 09:45, Sam Millman <sam.mill...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Is there a reason why you have $uuid field to the relatedid?
>
> > If not take it out and then you can do:
>
> > find({'Sub.RelatedId': "7bc745f9-77eb-4e7e-b3a3-1dfd0231545f"})
>

> >> "mongodb-user" group.>> To post to this group, send email tomongo...@googlegroups.com.

[Sam Millman]

Ah to get the subs out of that array with the same relatedid would have to be done client side atm.

You can get all root documents with the relatedid but for the min (until https://jira.mongodb.org/browse/SERVER-828) you cannot actually directly return only these subdocuments.

Is this a query you will perform often? It might be better to split the subcollection into a separate collection if it starts to slow down your system a little.

To post to this group, send email to mongod...@googlegroups.com.

[Sam Millman]

db.Commits.find({ "$elemMatch" : { "Sub.RelatedId" :
{"$uuid":"4fa8064f-349e-80b7-68de-af4c02c355be"}}})

Yea that wont work because $uuid is a special type.

Try elemMatch on the c# MongoId or ObjectId (not sure which it uses), sorry I cannot provide more code support I have not really programmed C# for a couple of years so I'm a little rusty.

[Steve Beaugé]

Sam, I know I'll get whole "parent" document as result and it's not a
problem, because all documents will contains at least one sub with the
correct relatedId, plus other sub with other relatedId.
The app will filter sub appropriately.

Splitting to a dedicated collection will send me to the start... I
actually wanted to avoid splitting to keep atomically inserting
several sub.

I can rephrase my current problem :

How to get all documents from the collection having at least on sub
with the corresponding relatedId ?

thx

On 8 nov, 11:15, Sam Millman <sam.mill...@gmail.com> wrote:
> Ah to get the subs out of that array with the same relatedid would have to
> be done client side atm.
>

> You can get all root documents with the relatedid but for the min (untilhttps://jira.mongodb.org/browse/SERVER-828) you cannot actually directly

> return only these subdocuments.
>
> Is this a query you will perform often? It might be better to split the
> subcollection into a separate collection if it starts to slow down your
> system a little.
>

> > "mongodb-user" group.> To post to this group, send email tomongo...@googlegroups.com.

> > To unsubscribe from this group, send email to>mongodb-user...@googlegroups.com.
> > For more
>

> ...
>
> plus de détails »

[Steve Beaugé]

You said : "Yea that wont work because $uuid is a special type.". Does
it mean elemMatch is not compatible with $uuid?



And for the C# code... I'll take it later. I'm using mongo client
console to test my request before translating it into C#.

On 8 nov, 11:18, Sam Millman <sam.mill...@gmail.com> wrote:
> db.Commits.find({ "$elemMatch" : { "Sub.RelatedId" :
> {"$uuid":"4fa8064f-349e-80b7-68de-af4c02c355be"}}})
>
> Yea that wont work because $uuid is a special type.
>
> Try elemMatch on the c# MongoId or ObjectId (not sure which it uses), sorry
> I cannot provide more code support I have not really programmed C# for a
> couple of years so I'm a little rusty.
>

> On 8 November 2011 10:15, Sam Millman <sam.mill...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Ah to get the subs out of that array with the same relatedid would have to
> > be done client side atm.
>
> > You can get all root documents with the relatedid but for the min (until
> >https://jira.mongodb.org/browse/SERVER-828) you cannot actually directly
> > return only these subdocuments.
>
> > Is this a query you will perform often? It might be better to split the
> > subcollection into a separate collection if it starts to slow down your
> > system a little.
>

> ...
>
> plus de détails »

[Sam Millman]

"And for the C# code... I'll take it later. I'm using mongo client

console to test my request before translating it into C#."

Ok kool but this link will help you see what you need to use:

http://www.mongodb.org/display/DOCS/CSharp+Driver+Tutorial#CSharpDriverTutorial-ObjectIdandBsonObjectId

Try this query:

find({ 'Sub.RelatedId": new ObjectId("4fa8064f-349e-80b7-68de-af4c02c355be") })

Hopefully that will fit your schema, sorry that I've been a bit dumb up until now.

To post to this group, send email to mongod...@googlegroups.com.

[Steve Beaugé]

The query you suggested is not working, because my Id is not an object
Id, but an uuid. It throw this error :

Tue Nov 08 11:34:58 Assertion: 10448:invalid object id: length
Tue Nov 08 11:34:58 Error: 10448 invalid object id: length (shell):1

I also tried this (because I'm searching into the sub array):

db.Commits.find( { "Sub" : { "$elemMatch" : { "RelatedId" : {"$uuid" :
"4fa8064f-349e-80b7-68de-af4c02c355be"}}}})

but not working. error: { "$err" : "invalid operator: $uuid", "code" :
10068 }

On 8 nov, 11:31, Sam Millman <sam.mill...@gmail.com> wrote:
> "And for the C# code... I'll take it later. I'm using mongo client
> console to test my request before translating it into C#."
>
> Ok kool but this link will help you see what you need to use:
>

> http://www.mongodb.org/display/DOCS/CSharp+Driver+Tutorial#CSharpDriv...

>
> Try this query:
>
> find({ 'Sub.RelatedId": new ObjectId("4fa8064f-349e-80b7-68de-af4c02c355be")
>
> })
>
> Hopefully that will fit your schema, sorry that I've been a bit dumb up
> until now.
>

> ...
>
> plus de détails »

[Sam Millman]

Ok the only thing I can think of is that some how you have been able to put in a field with a $ before it then.

Not sure how since $ is special type (used for $or $and and properties of object, hence I thought it was a ObjectId) so mongo cannot define the difference between the two.

Can you show us the code you used to insert these documents?

> ...
>
> plus de détails »

[Nat]

Try using new UUID(uuidstring) instead of { "$uuid" : uuidstring }

[Steve Beaugé]

I finally found this :

db.Commits.find( { "Sub" : { "$elemMatch" : { "Related" : new
BinData(3,"Ce6Tre8HdECo3R/KiBvTLQ==")}}})

Which is working. The guid is actually a bnary value, and the shell
didn't supported my construct. Using the binary it works. Fortunately,
the C# driver will handle the conversion for me.

Thanks a lot for the help! you help me to get it:)

On 8 nov, 11:40, Sam Millman <sam.mill...@gmail.com> wrote:
> Ok the only thing I can think of is that some how you have been able to put
> in a field with a $ before it then.
>
> Not sure how since $ is special type (used for $or $and and properties of
> object, hence I thought it was a ObjectId) so mongo cannot define the
> difference between the two.
>
> Can you show us the code you used to insert these documents?
>

> ...
>
> plus de détails »