pymongo simulating a SQL autoincrement primary key with server-side javascript

[Phlip]

Mongoes:

We need a field, revision_id, to trivially increment, on the server,
each time we add a record.

We could do this using a counter table, per...

http://www.mongodb.org/display/DOCS/Object+IDs#ObjectIDs-SequenceNumbers

...or we could just select the maximum revision_id, each time we
write. Regardless which way we do it, the binding between pymongo and
JavaScript does not seem to be working. Here's my feeb attempt:

from pymongo import Connection
db = Connection().some_database
db.Fog.drop()
fog_table = db.Fog

db.system_js.counter = ''' function() {
var rets = db.Fog.find( { }, { _id:0,
revision_id:1} ).sort( { revision_id: 1 } ).limit(-1);

if (rets.length > 0 && rets[0].revision_id)
return rets[0].revision_id + 1;
return 10001;
} '''

for x in range(1, 100):

fog_table.insert( { 'revision_id': db.system_js.counter(),
'_value': 'Yar_%i' % x } )

for record in fog_table.find():
print record

That blurts out a table full of broken revision_ids:

{u'_value': u'Yar_32', u'revision_id': 10001.0, u'_id':
ObjectId('4e...')}
{u'_value': u'Yar_33', u'revision_id': 10001.0, u'_id':
ObjectId('4e...')}
{u'_value': u'Yar_34', u'revision_id': 10001.0, u'_id':
ObjectId('4e...')}

Note this is only useful to us if _js.counter() evaluates on the
server, because we hope to have many clients posting data.

So where are we going wrong?

--
Phlip
http://bit.ly/ZeekLand

[Nat]

I guess "rets" is not really an array. You might want to use findOne function instead. It would be a lot easier. Note that javascript will not scale quite well. You might want to use a separated counter collection instead and querying it using findAndModify as suggested in the link.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Sridhar]

More information on findAndModify can be found at
http://www.mongodb.org/display/DOCS/findAndModify+Command. Also make
sure to select the new value in all cases so you can consistently get
a new id

[Phlip]

> > I guess "rets" is not really an array. You might want to use findOne function instead. It would be a lot easier. Note that javascript will not scale quite well. You might want to use a separated counter collection instead and querying it using findAndModify as suggested in the link.

I don't know how to call findOne with a max. (And googling for such
generic terms would be problematic at best!;)

Y'all have also not referenced my other question - how to make sure
the lookup happens atomically, at insert time, on the server. Not just
before insert time, when the {} is filling up.

[Phlip]

I will guess at the answer - because findAndModify is itself atomic, a
separate table to hold the primary key's high water mark is still
efficient, and does not need to happen at insert time.

So, unless someone wants to stop me and suggest a better system,
findAndModify it is!

[Nat]

ok... findOne seems to be a bit limiting.... try something like

db.system_js.counter = ''' function() { 
                        var rets = db.Fog.find( { }, { _id:0, 
revision_id:1} ).sort( { revision_id: 1 } ).limit(-1); 

                        if (rets.hasNext()) 
                        {
                              var ret = rets.next();     
                              if (ret.revision_id) {

                                  return rets[0].revision_id + 1; 

                              } else {
                                  return 10001;
                              }
                        }
                        return 10001; 
                     } ''' 


Anyway.... using $inc with findAndModify is probably the way to go!

[Laurent Payot]

Hi Philip,

I could make autoincrement work with pymongo using the counter method you previously mentioned (http://www.mongodb.org/display/DOCS/Object+IDs#ObjectIDs-SequenceNumbers)

I simply do:

db.test_col.insert({'_id': db.eval("return counter('test_col');"), 'foo': 'bar'})

As you can see I have to use "eval" to retrieve the new _id, so I make two requests instead of one and this freakin' eval feature has a write lock and is incompatible with sharding.

Could you finally find a 'clean' way to autoincrement a primary key serverside? That's a big issue for me too.

[Scott Hernandez]

Do the findAndModify from python, and use the resulting id with the insert.

Do not use db.eval.

The example code you reference shows the two things done in the same
language (client), not at the server.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/wXsuT7lLyTgJ.

[Laurent]

Ok, I could finally make findAndModify work with pymongo with the following syntax:

int(db.counters.find_and_modify(query= {"_id": "test_col"}, update={"$inc": {"next":1}}, new=True, upsert=True)['next'])