Handling Mongo's ObjectId type

[Geoff Schmidt]

Right now Meteor does a poor job of handling Mongo databases that contain the native Mongo ObjectId type. See for example Xavier's ticket 61:

https://github.com/meteor/meteor/issues/61

Below is a thread that Glasser, Matt, and I were having. I thought we should move discussion to the list.

---------- Forwarded message ----------
From: Geoff Schmidt <gsch...@meteor.com>
Date: Fri, Sep 28, 2012 at 6:50 PM
Subject: Re: One relevant question re object IDs
To: David Glasser <gla...@meteor.com>
Cc: Matt DeBergalis <ma...@meteor.com>

You do have a way of getting right to the heart of things :)

Matt and I discussed this and we agree that the only option is to support ObjectId at the DDP level.

Specifically, I propose a new abstract data format, EJSON, which is just like JSON except it has three new types:

* "Binary" (an octet sequence)

* Date (with time -- details TBD)

* User-defined record (comprising a user-defined tag string such as "objectid", "latlng", or "complex", plus a payload that may be any EJSON value, but which is most likely an object.) The meaning of these records is by arrangement between the sender and the receiver. From the perspective of EJSON, it's just a distinct type that's defined by a pair (string, value).

And I propose an encoding of EJSON into JSON as follows:

- Binary maps to {$type: "octets", data: encodedData} where encodedData is the octets as a string (octets 0-255 are mapped to codepoints 0-255)

- Date maps to {$type: "date", .. date fields tbd .. }

- User-defined maps to {$type: "%tag", data: userDefinedData}, eg {$type: "%objectid", data: "123456789abcdef"}

- Objects that contain a $type attribute map to {$type: "object", value: originalJsonObject}

- All other EJSON values map directly onto the corresponding JSON value

Our livedata package (which will eventually be split in two and renamed 'ddp-client' and 'ddp-server') should map the binary type onto TypedArray, the date type onto Date, the user-defined type named "objectid" onto MongoObjectId, and all other user-defined types onto CustomType. mongo-livedata should maps the first three of those to the appropriate mongo values, and throw an exception if it encounters CustomType.

Finally, once we have this, we probably should drop the UUID strings and just generate random MongoObjectIds. The only reason we didn't do this before is that we didn't have an ObjectId type.

However, notice that although Mongo allows ObjectId's to be any sequence of 12 octets, Mongo usually generates ObjectIds that have a particular structure which includes the current unix time.

http://api.mongodb.org/wiki/current/Object%20IDs.html#ObjectIDs-TheBSONObjectIdDatatype

http://api.mongodb.org/wiki/current/Optimizing%20Object%20IDs.html

They do this so that ObjectIds of new objects are in "roughly ascending order", meaning that if you're just inserting a bunch of records, you don't have to keep the entire _id index in memory, just the rightmost part of the btree. Maybe in Meteor 3.0 we will care about such niceties and lease blocks of ids to clients similar to how mongo implicitly leases blocks of ids to servers.

On Fri, Sep 28, 2012 at 2:19 PM, David Glasser <gla...@meteor.com> wrote:

Can users pass Object IDs as args to their own methods or return them? Or use them as fields (primary key or otherwise) on custom publishers?

If yes, and I suspect it is yes, then we need to solve this at the live data/DDP protocol level rather than the livedata-mongo level.

[David Glasser]

That sounds good.

What does the DDP ID look like, though? Ie, the id field of the msg:data message. Are you proposing that it may now be any EJSON object? That's fine by me, but you were suggesting when we talked today that such ids may likely be used in URLs. (By the core system, or were you just thinking by apps?) Not sure how pretty that ends up.

When and who should do this? (At least implementing enough to support object ID).

--
You received this message because you are subscribed to the Google Groups "meteor-core" group.
To post to this group, send email to meteo...@googlegroups.com.
To unsubscribe from this group, send email to meteor-core...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/meteor-core?hl=en.

[Tom Coleman]

Perhaps date can be a custom type too? (date, 1348887460489) (where 1348887460489 is the the timestamp).

[David Greenspan]

If object _ids aren't strings on the client, it's going to make a lot of code messy (in minimongo and apps).  No more maps keyed on ids.  I don't know why Mongo has ObjectId objects, but can we just map them bidirectionally to strings in our mongo driver?

I do like the wire protocol that isn't constrained to JSONable objects, and I assume it would be useful.

-- David

[David Glasser]

One option is that we may need to implement a javascript map class
whose keys are (immutable) JSON/EJSON objects and use that in a lot of
places instead just an object. The syntax would be worse, but it's
already the case that Javascript objects make poor maps (eg, needing
to use hasOwnProperty/_.has).