You do have a way of getting right to the heart of things :)
Matt and I discussed this and we agree that the only option is to support
ObjectId at the DDP level.
Specifically, I propose a new abstract data format, EJSON, which is just
like JSON except it has three new types:
* "Binary" (an octet sequence)
* Date (with time -- details TBD)
* User-defined record (comprising a user-defined tag string such as
"objectid", "latlng", or "complex", plus a payload that may be any EJSON
value, but which is most likely an object.) The meaning of these records is
by arrangement between the sender and the receiver. From the perspective of
EJSON, it's just a distinct type that's defined by a pair (string, value).
And I propose an encoding of EJSON into JSON as follows:
- Binary maps to {$type: "octets", data: encodedData} where encodedData is
the octets as a string (octets 0-255 are mapped to codepoints 0-255)
- Date maps to {$type: "date", .. date fields tbd .. }
- User-defined maps to {$type: "%tag", data: userDefinedData}, eg {$type:
"%objectid", data: "123456789abcdef"}
- Objects *that contain a $type attribute* map to {$type: "object", value:
originalJsonObject}
- All other EJSON values map directly onto the corresponding JSON value
Our livedata package (which will eventually be split in two and renamed
'ddp-client' and 'ddp-server') should map the binary type onto TypedArray,
the date type onto Date, the user-defined type named "objectid" onto
MongoObjectId, and all other user-defined types onto CustomType.
mongo-livedata should maps the first three of those to the appropriate
mongo values, and throw an exception if it encounters CustomType.
Finally, once we have this, we probably should drop the UUID strings and
just generate random MongoObjectIds. The only reason we didn't do this
before is that we didn't have an ObjectId type.
They do this so that ObjectIds of new objects are in "roughly ascending
order", meaning that if you're just inserting a bunch of records, you don't
have to keep the entire _id index in memory, just the rightmost part of the
btree. Maybe in Meteor 3.0 we will care about such niceties and lease
blocks of ids to clients similar to how mongo implicitly leases blocks of
ids to servers.
On Fri, Sep 28, 2012 at 2:19 PM, David Glasser <glas...@meteor.com> wrote:
> Can users pass Object IDs as args to their own methods or return them? Or
> use them as fields (primary key or otherwise) on custom publishers?
> If yes, and I suspect it is yes, then we need to solve this at the live
> data/DDP protocol level rather than the livedata-mongo level.
What does the DDP ID look like, though? Ie, the id field of the msg:data
message. Are you proposing that it may now be any EJSON object? That's fine
by me, but you were suggesting when we talked today that such ids may
likely be used in URLs. (By the core system, or were you just thinking by
apps?) Not sure how pretty that ends up.
When and who should do this? (At least implementing enough to support
object ID).
On Sep 28, 2012 7:07 PM, "Geoff Schmidt" <gschm...@meteor.com> wrote:
> Below is a thread that Glasser, Matt, and I were having. I thought we
> should move discussion to the list.
> ---------- Forwarded message ----------
> From: Geoff Schmidt <gschm...@meteor.com>
> Date: Fri, Sep 28, 2012 at 6:50 PM
> Subject: Re: One relevant question re object IDs
> To: David Glasser <glas...@meteor.com>
> Cc: Matt DeBergalis <m...@meteor.com>
> You do have a way of getting right to the heart of things :)
> Matt and I discussed this and we agree that the only option is to support
> ObjectId at the DDP level.
> Specifically, I propose a new abstract data format, EJSON, which is just
> like JSON except it has three new types:
> * "Binary" (an octet sequence)
> * Date (with time -- details TBD)
> * User-defined record (comprising a user-defined tag string such as
> "objectid", "latlng", or "complex", plus a payload that may be any EJSON
> value, but which is most likely an object.) The meaning of these records is
> by arrangement between the sender and the receiver. From the perspective of
> EJSON, it's just a distinct type that's defined by a pair (string, value).
> And I propose an encoding of EJSON into JSON as follows:
> - Binary maps to {$type: "octets", data: encodedData} where encodedData is
> the octets as a string (octets 0-255 are mapped to codepoints 0-255)
> - Date maps to {$type: "date", .. date fields tbd .. }
> - User-defined maps to {$type: "%tag", data: userDefinedData}, eg {$type:
> "%objectid", data: "123456789abcdef"}
> - Objects *that contain a $type attribute* map to {$type: "object",
> value: originalJsonObject}
> - All other EJSON values map directly onto the corresponding JSON value
> Our livedata package (which will eventually be split in two and renamed
> 'ddp-client' and 'ddp-server') should map the binary type onto TypedArray,
> the date type onto Date, the user-defined type named "objectid" onto
> MongoObjectId, and all other user-defined types onto CustomType.
> mongo-livedata should maps the first three of those to the appropriate
> mongo values, and throw an exception if it encounters CustomType.
> Finally, once we have this, we probably should drop the UUID strings and
> just generate random MongoObjectIds. The only reason we didn't do this
> before is that we didn't have an ObjectId type.
> However, notice that although Mongo allows ObjectId's to be any sequence
> of 12 octets, Mongo usually generates ObjectIds that have a particular
> structure which includes the current unix time.
> They do this so that ObjectIds of new objects are in "roughly ascending
> order", meaning that if you're just inserting a bunch of records, you don't
> have to keep the entire _id index in memory, just the rightmost part of the
> btree. Maybe in Meteor 3.0 we will care about such niceties and lease
> blocks of ids to clients similar to how mongo implicitly leases blocks of
> ids to servers.
> On Fri, Sep 28, 2012 at 2:19 PM, David Glasser <glas...@meteor.com> wrote:
>> Can users pass Object IDs as args to their own methods or return them? Or
>> use them as fields (primary key or otherwise) on custom publishers?
>> If yes, and I suspect it is yes, then we need to solve this at the live
>> data/DDP protocol level rather than the livedata-mongo level.
> --
> You received this message because you are subscribed to the Google Groups
> "meteor-core" group.
> To post to this group, send email to meteor-core@googlegroups.com.
> To unsubscribe from this group, send email to
> meteor-core+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/meteor-core?hl=en.
> Below is a thread that Glasser, Matt, and I were having. I thought we should move discussion to the list.
> ---------- Forwarded message ----------
> From: Geoff Schmidt <gschm...@meteor.com (mailto:gschm...@meteor.com)>
> Date: Fri, Sep 28, 2012 at 6:50 PM
> Subject: Re: One relevant question re object IDs
> To: David Glasser <glas...@meteor.com (mailto:glas...@meteor.com)>
> Cc: Matt DeBergalis <m...@meteor.com (mailto:m...@meteor.com)>
> You do have a way of getting right to the heart of things :)
> Matt and I discussed this and we agree that the only option is to support ObjectId at the DDP level.
> Specifically, I propose a new abstract data format, EJSON, which is just like JSON except it has three new types:
> * "Binary" (an octet sequence)
> * Date (with time -- details TBD)
> * User-defined record (comprising a user-defined tag string such as "objectid", "latlng", or "complex", plus a payload that may be any EJSON value, but which is most likely an object.) The meaning of these records is by arrangement between the sender and the receiver. From the perspective of EJSON, it's just a distinct type that's defined by a pair (string, value).
> And I propose an encoding of EJSON into JSON as follows:
> - Binary maps to {$type: "octets", data: encodedData} where encodedData is the octets as a string (octets 0-255 are mapped to codepoints 0-255)
> - Date maps to {$type: "date", .. date fields tbd .. }
> - User-defined maps to {$type: "%tag", data: userDefinedData}, eg {$type: "%objectid", data: "123456789abcdef"}
> - Objects that contain a $type attribute map to {$type: "object", value: originalJsonObject}
> - All other EJSON values map directly onto the corresponding JSON value
> Our livedata package (which will eventually be split in two and renamed 'ddp-client' and 'ddp-server') should map the binary type onto TypedArray, the date type onto Date, the user-defined type named "objectid" onto MongoObjectId, and all other user-defined types onto CustomType. mongo-livedata should maps the first three of those to the appropriate mongo values, and throw an exception if it encounters CustomType.
> Finally, once we have this, we probably should drop the UUID strings and just generate random MongoObjectIds. The only reason we didn't do this before is that we didn't have an ObjectId type.
> They do this so that ObjectIds of new objects are in "roughly ascending order", meaning that if you're just inserting a bunch of records, you don't have to keep the entire _id index in memory, just the rightmost part of the btree. Maybe in Meteor 3.0 we will care about such niceties and lease blocks of ids to clients similar to how mongo implicitly leases blocks of ids to servers.
> On Fri, Sep 28, 2012 at 2:19 PM, David Glasser <glas...@meteor.com (mailto:glas...@meteor.com)> wrote:
> > Can users pass Object IDs as args to their own methods or return them? Or use them as fields (primary key or otherwise) on custom publishers?
> > If yes, and I suspect it is yes, then we need to solve this at the live data/DDP protocol level rather than the livedata-mongo level.
> --
> You received this message because you are subscribed to the Google Groups "meteor-core" group.
> To post to this group, send email to meteor-core@googlegroups.com (mailto:meteor-core@googlegroups.com).
> To unsubscribe from this group, send email to meteor-core+unsubscribe@googlegroups.com (mailto:meteor-core+unsubscribe@googlegroups.com).
> For more options, visit this group at http://groups.google.com/group/meteor-core?hl=en.
If object _ids aren't strings on the client, it's going to make a lot of
code messy (in minimongo and apps). No more maps keyed on ids. I don't
know why Mongo has ObjectId objects, but can we just map them
bidirectionally to strings in our mongo driver?
I do like the wire protocol that isn't constrained to JSONable objects, and
I assume it would be useful.
On Fri, Sep 28, 2012 at 7:26 PM, David Glasser <glas...@meteor.com> wrote:
> That sounds good.
> What does the DDP ID look like, though? Ie, the id field of the msg:data
> message. Are you proposing that it may now be any EJSON object? That's fine
> by me, but you were suggesting when we talked today that such ids may
> likely be used in URLs. (By the core system, or were you just thinking by
> apps?) Not sure how pretty that ends up.
> When and who should do this? (At least implementing enough to support
> object ID).
> On Sep 28, 2012 7:07 PM, "Geoff Schmidt" <gschm...@meteor.com> wrote:
>> Right now Meteor does a poor job of handling Mongo databases that contain
>> the native Mongo ObjectId type. See for example Xavier's ticket 61:
>> Below is a thread that Glasser, Matt, and I were having. I thought we
>> should move discussion to the list.
>> ---------- Forwarded message ----------
>> From: Geoff Schmidt <gschm...@meteor.com>
>> Date: Fri, Sep 28, 2012 at 6:50 PM
>> Subject: Re: One relevant question re object IDs
>> To: David Glasser <glas...@meteor.com>
>> Cc: Matt DeBergalis <m...@meteor.com>
>> You do have a way of getting right to the heart of things :)
>> Matt and I discussed this and we agree that the only option is to support
>> ObjectId at the DDP level.
>> Specifically, I propose a new abstract data format, EJSON, which is just
>> like JSON except it has three new types:
>> * "Binary" (an octet sequence)
>> * Date (with time -- details TBD)
>> * User-defined record (comprising a user-defined tag string such as
>> "objectid", "latlng", or "complex", plus a payload that may be any EJSON
>> value, but which is most likely an object.) The meaning of these records is
>> by arrangement between the sender and the receiver. From the perspective of
>> EJSON, it's just a distinct type that's defined by a pair (string, value).
>> And I propose an encoding of EJSON into JSON as follows:
>> - Binary maps to {$type: "octets", data: encodedData} where encodedData
>> is the octets as a string (octets 0-255 are mapped to codepoints 0-255)
>> - Date maps to {$type: "date", .. date fields tbd .. }
>> - User-defined maps to {$type: "%tag", data: userDefinedData}, eg {$type:
>> "%objectid", data: "123456789abcdef"}
>> - Objects *that contain a $type attribute* map to {$type: "object",
>> value: originalJsonObject}
>> - All other EJSON values map directly onto the corresponding JSON value
>> Our livedata package (which will eventually be split in two and renamed
>> 'ddp-client' and 'ddp-server') should map the binary type onto TypedArray,
>> the date type onto Date, the user-defined type named "objectid" onto
>> MongoObjectId, and all other user-defined types onto CustomType.
>> mongo-livedata should maps the first three of those to the appropriate
>> mongo values, and throw an exception if it encounters CustomType.
>> Finally, once we have this, we probably should drop the UUID strings and
>> just generate random MongoObjectIds. The only reason we didn't do this
>> before is that we didn't have an ObjectId type.
>> However, notice that although Mongo allows ObjectId's to be any sequence
>> of 12 octets, Mongo usually generates ObjectIds that have a particular
>> structure which includes the current unix time.
>> They do this so that ObjectIds of new objects are in "roughly ascending
>> order", meaning that if you're just inserting a bunch of records, you don't
>> have to keep the entire _id index in memory, just the rightmost part of the
>> btree. Maybe in Meteor 3.0 we will care about such niceties and lease
>> blocks of ids to clients similar to how mongo implicitly leases blocks of
>> ids to servers.
>> On Fri, Sep 28, 2012 at 2:19 PM, David Glasser <glas...@meteor.com>wrote:
>>> Can users pass Object IDs as args to their own methods or return them?
>>> Or use them as fields (primary key or otherwise) on custom publishers?
>>> If yes, and I suspect it is yes, then we need to solve this at the live
>>> data/DDP protocol level rather than the livedata-mongo level.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "meteor-core" group.
>> To post to this group, send email to meteor-core@googlegroups.com.
>> To unsubscribe from this group, send email to
>> meteor-core+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/meteor-core?hl=en.
> --
> You received this message because you are subscribed to the Google Groups
> "meteor-core" group.
> To post to this group, send email to meteor-core@googlegroups.com.
> To unsubscribe from this group, send email to
> meteor-core+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/meteor-core?hl=en.
One option is that we may need to implement a javascript map class
whose keys are (immutable) JSON/EJSON objects and use that in a lot of
places instead just an object. The syntax would be worse, but it's
already the case that Javascript objects make poor maps (eg, needing
to use hasOwnProperty/_.has).
On Fri, Sep 28, 2012 at 10:05 PM, David Greenspan <da...@meteor.com> wrote:
> If object _ids aren't strings on the client, it's going to make a lot of
> code messy (in minimongo and apps). No more maps keyed on ids. I don't
> know why Mongo has ObjectId objects, but can we just map them
> bidirectionally to strings in our mongo driver?
> I do like the wire protocol that isn't constrained to JSONable objects, and
> I assume it would be useful.
> -- David
> On Fri, Sep 28, 2012 at 7:26 PM, David Glasser <glas...@meteor.com> wrote:
>> That sounds good.
>> What does the DDP ID look like, though? Ie, the id field of the msg:data
>> message. Are you proposing that it may now be any EJSON object? That's fine
>> by me, but you were suggesting when we talked today that such ids may likely
>> be used in URLs. (By the core system, or were you just thinking by apps?)
>> Not sure how pretty that ends up.
>> When and who should do this? (At least implementing enough to support
>> object ID).
>>> Below is a thread that Glasser, Matt, and I were having. I thought we
>>> should move discussion to the list.
>>> ---------- Forwarded message ----------
>>> From: Geoff Schmidt <gschm...@meteor.com>
>>> Date: Fri, Sep 28, 2012 at 6:50 PM
>>> Subject: Re: One relevant question re object IDs
>>> To: David Glasser <glas...@meteor.com>
>>> Cc: Matt DeBergalis <m...@meteor.com>
>>> You do have a way of getting right to the heart of things :)
>>> Matt and I discussed this and we agree that the only option is to support
>>> ObjectId at the DDP level.
>>> Specifically, I propose a new abstract data format, EJSON, which is just
>>> like JSON except it has three new types:
>>> * "Binary" (an octet sequence)
>>> * Date (with time -- details TBD)
>>> * User-defined record (comprising a user-defined tag string such as
>>> "objectid", "latlng", or "complex", plus a payload that may be any EJSON
>>> value, but which is most likely an object.) The meaning of these records is
>>> by arrangement between the sender and the receiver. From the perspective of
>>> EJSON, it's just a distinct type that's defined by a pair (string, value).
>>> And I propose an encoding of EJSON into JSON as follows:
>>> - Binary maps to {$type: "octets", data: encodedData} where encodedData
>>> is the octets as a string (octets 0-255 are mapped to codepoints 0-255)
>>> - Date maps to {$type: "date", .. date fields tbd .. }
>>> - User-defined maps to {$type: "%tag", data: userDefinedData}, eg {$type:
>>> "%objectid", data: "123456789abcdef"}
>>> - Objects that contain a $type attribute map to {$type: "object", value:
>>> originalJsonObject}
>>> - All other EJSON values map directly onto the corresponding JSON value
>>> Our livedata package (which will eventually be split in two and renamed
>>> 'ddp-client' and 'ddp-server') should map the binary type onto TypedArray,
>>> the date type onto Date, the user-defined type named "objectid" onto
>>> MongoObjectId, and all other user-defined types onto CustomType.
>>> mongo-livedata should maps the first three of those to the appropriate mongo
>>> values, and throw an exception if it encounters CustomType.
>>> Finally, once we have this, we probably should drop the UUID strings and
>>> just generate random MongoObjectIds. The only reason we didn't do this
>>> before is that we didn't have an ObjectId type.
>>> However, notice that although Mongo allows ObjectId's to be any sequence
>>> of 12 octets, Mongo usually generates ObjectIds that have a particular
>>> structure which includes the current unix time.
>>> They do this so that ObjectIds of new objects are in "roughly ascending
>>> order", meaning that if you're just inserting a bunch of records, you don't
>>> have to keep the entire _id index in memory, just the rightmost part of the
>>> btree. Maybe in Meteor 3.0 we will care about such niceties and lease blocks
>>> of ids to clients similar to how mongo implicitly leases blocks of ids to
>>> servers.
>>> On Fri, Sep 28, 2012 at 2:19 PM, David Glasser <glas...@meteor.com>
>>> wrote:
>>>> Can users pass Object IDs as args to their own methods or return them?
>>>> Or use them as fields (primary key or otherwise) on custom publishers?
>>>> If yes, and I suspect it is yes, then we need to solve this at the live
>>>> data/DDP protocol level rather than the livedata-mongo level.
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "meteor-core" group.
>>> To post to this group, send email to meteor-core@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> meteor-core+unsubscribe@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/meteor-core?hl=en.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "meteor-core" group.
>> To post to this group, send email to meteor-core@googlegroups.com.
>> To unsubscribe from this group, send email to
>> meteor-core+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/meteor-core?hl=en.
> --
> You received this message because you are subscribed to the Google Groups
> "meteor-core" group.
> To post to this group, send email to meteor-core@googlegroups.com.
> To unsubscribe from this group, send email to
> meteor-core+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/meteor-core?hl=en.
