Handling ObjectIds in Reference Array with JSON

[Bradley Locke]

Hello all,

I would first like to admit that I am still fairly new to clojure, mongodb, and monger.  I've tried too search around for the answer to my question and have had no luck.

I am creating a web application that is AngularJS/Luminus/Monger/MongoDB.  I understand that MongoDB uses ObjectIds for _ids which I don't completely understand the reasoning for, but am willing to accept it.  (Note: I know they can be used for timestamps and are good at being unique and ordering results)  They seem to be difficult to work with for REST/JSON web applications, at least to me.  I've honestly not heard others complaining about this problem, so I may just be thinking about it all wrong.

The current issue I'm trying to work with is how to deserialize my JSON object that is being POSTed back from my front end into the correct format for storing in the database.  I'm manually having to check for the _id field and running monger.conversion/to-object-id, which I feel is a headache, but the bigger issue is if one document has an array of references to other documents (many-to-many).  I am able to retrieve it from the database, serialize it such that JSON can handle passing it to my front end, however, when I retrieve a modified version back, I don't have an easy way to know what should be converted back to ObjectIds.  I could hardcode looking for the field that stores this array and run through and convert it, but that just doesn't seem like the ideal way to do it.

I've tried searching for information about strict mode, as I found that it handles JSONifying data by using $oid fields to reference, but from what I can tell, that is through MongoDBs REST API and only works for retrieval, not for inserting.

I also see some people going the route of not using ObjectIds as their _ids and opting for strings or numbers.  It seems like an acceptable approach according to Mongo, but just not their preferred way.  And if you forget to add an _id they will create one for you using ObjectIds which could just throw the whole thing off.

TL;DR

My question is this: How do you all handle working with deserializing JSON data correctly for a document that has an array of referencing document ObjectIds?

Thanks!

[Michael Klishin]

2014-06-14 7:03 GMT+04:00 Bradley Locke <bsl...@gmail.com>:

The current issue I'm trying to work with is how to deserialize my JSON object that is being POSTed back from my front end into the correct format for storing in the database.  I'm manually having to check for the _id field and running monger.conversion/to-object-id, which I feel is a headache, but the bigger issue is if one document has an array of references to other documents (many-to-many).  I am able to retrieve it from the database, serialize it such that JSON can handle passing it to my front end, however, when I retrieve a modified version back, I don't have an easy way to know what should be converted back to ObjectIds.  I could hardcode looking for the field that stores this array and run through and convert it, but that just doesn't seem like the ideal way to do it.

This question is not specific to Monger. The answer you'll hear from many MongoDB users is "with MongoDB you need

to denormalize and embed, not reference". It's not particularly difficult to map a list of ids to a list of documents loaded,

just remember that clojure.core/map is lazy. Things can get more painful when you have arbitrarily nested documents

but even then you can assume that fields named _ids or ending in _ids need to be "resolved" (mapped with a function

that loads documents).

Most practices that work with any other driver should be applicable to Monger.

--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

[Bradley Locke]

Thanks for the response Michael.

I understand that this isn't a direct question regarding Monger.  I believe it's more of a general question regarding Clojure (or other dynamically typed languages) and how it works with data in MongoDB.  Monger is kind of the glue between the two, so I figured it was a good place to ask the question to other Monger users.

I had originally thought about denormalizing my data, but my use case would be quite difficult with denormalized data.  I will have items of type B that store each store a reference to (or embed items of) type A.  The caveat is that these type A objects can be used by different type B objects.  Type B can add/remove references to A, and type A objects can be updated.  If I denormalized, I would still need a way to know that when a type A object is changed that is shared by different type B objects that I need to go and manually update each type B record to do so.

I came across this link on MongoDB's website:

http://docs.mongodb.org/manual/reference/database-references/#DatabaseReferences-DBRef

It mentions that most cases denormalizing makes sense, but there are times when it doesn't.  I'm feeling like mine is a case where it doesn't.  I also recognize that document databases are not useful in every scenario, however, I like the flexibility it provides by allowing me to have some type A objects that have certain fields in them and others that have completely different fields.  They do share some similar properties, which is why they are grouped under A.

Would there be any downside to me using something else besides ObjectIds?  I was thinking maybe I store the IDs as strings that can translate to ObjectIds.  (-> (ObjectId.) (clojure.core/str)).  This way I'd get the uniqueness of ObjectIds for inserting as well as the benefits of using the id as a creation timestamp, but I don't have to worry about serializing/deserializing each time I pass between my front and back end.

[Michael Klishin]

2014-06-15 5:48 GMT+04:00 Bradley Locke <bsl...@gmail.com>:

I came across this link on MongoDB's website:

http://docs.mongodb.org/manual/reference/database-references/#DatabaseReferences-DBRef

AFAIR DB refs were deprecated at some point.

 

It mentions that most cases denormalizing makes sense, but there are times when it doesn't.  I'm feeling like mine is a case where it doesn't.  I also recognize that document databases are not useful in every scenario, however, I like the flexibility it provides by allowing me to have some type A objects that have certain fields in them and others that have completely different fields.  They do share some similar properties, which is why they are grouped under A.

Would there be any downside to me using something else besides ObjectIds?  I was thinking maybe I store the IDs as strings that can translate to ObjectIds.  (-> (ObjectId.) (clojure.core/str)).  This way I'd get the uniqueness of ObjectIds for inserting as well as the benefits of using the id as a creation timestamp, but I don't have to worry about serializing/deserializing each time I pass between my front and back end.

ObjectIds can be sorted (by timestamp, counter) and grouped and thus is a good candidate for a primary key but you can use anything

if you don't care about things like shards getting out of balance. However, your problem seems to be with "how do I automatically

load documents based on a field naming convention", so avoiding object id doesn't save you much effort: just one fewer

map operation. The hardest part is arbitrary nesting of maps, not object ids.