Best way to use custom ID's?

[midnightcowboy]

I'm interested in using a different ID format for documents instead of the default BSON::ObjectId format. What would be the best way to implement this? I already tried something like

key :_id, String, :required => true, :default => lambda { generate_custom_id }

or

before_create do

  self._id = generate_custom_id

end

and even though it seems to work, I'm wondering if there are nicer ways of doing this. The official Mongo Ruby driver supports custom ID formats by supplying a primary key factory, see here: http://api.mongodb.org/ruby/current/#Primary_Keys. I've looked through MongoMapper's code but it seems like there is no support for it.

Any thoughts?

Cheers

[John Nunemaker]

Correct, MongoMapper doesn't have any primary key factories right now. Overriding :_id and setting a default is probably the best way. I'd probably do SomeKeyFactory.next instead of a method on the model.

--
You received this message because you are subscribed to the Google
Groups "MongoMapper" group.
For more options, visit this group at
http://groups.google.com/group/mongomapper?hl=en?hl=en

[Jamie Orchard-Hays]

I like your lambda method as you'll always have an id, which is the way MongoMapper works with regular Mongo ids.

[Brian Hempel]

I tried doing sequential id's a ways back, and I think the dirty plugin called the key default method every time an object was loaded from the database. As a consequence, our sequence was incremented way too fast...just something to keep in mind.

Brian

[John Nunemaker]

If that happens and you can make a simple test case/example, create an issue and someone will get it fixed. Seems like an easy one.

[midnightcowboy]

Yes I also noticed that the default method is called every time an object is loaded, which seemed inefficient. That's why I was looking for other ways than using key default.

[Brian Hempel]

After perusing code, here's what I've learned:

Primary keys are supposed to provide a database-wide way to define primary keys.

When you insert documents into Mongo, the Ruby driver runs every single doc through the PK factory to ensure each doc hash has a primary key set.

Inserted documents are run through the pk_factory: https://github.com/mongodb/mongo-ruby-driver/blob/master/lib/mongo/collection.rb#L351

The default, BSON::ObjectId pk factory: https://github.com/mongodb/mongo-ruby-driver/blob/master/lib/bson/types/object_id.rb#L92-94

Mongo's primary key factories are of somewhat limited use. Ostensibly, you aren't supposed to change the pk_factory once you set it, so the Ruby driver complains if you try to set it twice for on a database.  Also, it's set per-database, not per-collection. I'm guessing those constraints are overzealous.  With some meta-programming you could change the pk_factory per collection and I doubt you'll see any ill effects.

But in MongoMapper, the _id key is set before the document hits the Ruby driver.  So the pk_factory doesn't do anything because _id is already set.

I don't see any advantage to using pk_factories over MM's _id defaults, nor of reworking MM to provide an interface to set the pk factory.

MongoMapper code:

A doc's _id is first set on save as to_mongo hits the default method for _id:

https://github.com/jnunemaker/mongomapper/blob/master/lib/mongo_mapper/plugins/keys.rb#L11

https://github.com/jnunemaker/mongomapper/blob/master/lib/mongo_mapper/plugins/keys.rb#L191-194 # oops, should probably be key.name == :_id instead of key.type == ObjectId

https://github.com/jnunemaker/mongomapper/blob/master/lib/mongo_mapper/plugins/keys.rb#L241-243

https://github.com/jnunemaker/mongomapper/blob/master/lib/mongo_mapper/plugins/keys.rb#L288-294

https://github.com/jnunemaker/mongomapper/blob/master/lib/mongo_mapper/plugins/keys/key.rb#L28-44

(Diff regarding key.name == :_id, has to do w/not saving nil attributes: https://github.com/jnunemaker/mongomapper/compare/d46ed59c5213cfe2633bfee544c1f8c2c5a1020d...8702fec636e6f65a7fb0cb07d9effbccd881f7fe#diff-20)

Brian

[midnightcowboy]

Something I'm still not sure about is how to ensure that ID's are unique when using custom IDs. I once did some quick tests where I set the default to a static value, e.g.

key :_id, String, :required => true, :default => 'abcde'

When creating the second document I expected to get an error telling me that a document with the ID already exists (since the _id field has an index by default), but surprisingly the second doc simply overwrote the first one. Is there any protection in Mongo or MongoMapper to ensure unique IDs? I guess I could just set :unique => true for the key, but as far as I remember the :unique validation doesn't work for embedded docs.

[John Nunemaker]

MongoMapper defaults to the safe value of the driver, which is false. This means that errors don't raise. 

You can read more about the safe option here:

http://api.mongodb.org/ruby/1.6.1/Mongo/Connection.html#initialize-instance_method

Also, here is an MM safe example:

https://github.com/jnunemaker/mongomapper/blob/master/examples/safe.rb

The plugin:

https://github.com/jnunemaker/mongomapper/blob/master/lib/mongo_mapper/plugins/safe.rb

The tests:

https://github.com/jnunemaker/mongomapper/blob/master/test/functional/test_safe.rb

[midnightcowboy]

Hi John,

thanks for pointing me in the right direction.

My app was actually running Mongo in safe mode already, so I've done some testing with the safe example in MongoMapper. When I run the safe.rb file, it works as expected. However when I modify it slightly into this https://gist.github.com/2157677 then the behaviour is somewhat unexpected. I added two keys, and the :_id always has the same value. I also added a :unique index on :_id.

When I run the file, it keeps overwriting the record, rather than raising an exception for duplicate IDs — even when using User.create. My guess is, it has to do with Brian's observation that in MongoMapper, the _id is set before it hits the Ruby driver. Then likely the Ruby driver interprets it as an update to an existing record, rather than a create. I can also see that in the log files, where the Ruby driver performs update instead of insert, even for new documents. When I remove the manual :_id key from my models and let it use ObjectIds, it's still the same.

I'm not sure if this is a bug or expected behaviour, but for now it seems the result for using custom IDs with MM is, you always have to check manually whether a record with that ID already exists, to prevent existing ones from being overwritten.

Any ideas?

[John Nunemaker]

Actually I believe it is because we use save instead of insert/update. I think only insert with safe raises for duplicate id. 

[Jon Kern]

yea, in past apps (like 15 years ago), where i needed to, i created a simple KeyTable sort of gizmo that:

• returned the next key for a given collection
• returned the next n keys if you needed more
• tracked that "last used" key for each connection
  
• (and when ints were small, had a way to recycle unused keys)
  

of course, that sort of gizmo requires lots of care as you scale to multiple instances, etc.

jon

blog: http://technicaldebt.com
twitter: http://twitter.com/JonKernPA

John Nunemaker said the following on 3/21/12 12:16 PM:

[Shem Magnezi]

Just be aware that this method will be called only when you're using "create" to insert this doc to db. if you're using "new" or something (for example- for bulk insert) this won't be called.

Any idea what to do in this case instead of setting id by yourself?