Advice on patterns for reusing mongodb calls

[jbk]

I'm working on a web application using node-mongodb-native and
express.js. There are a number of URLs that perform similar actions:
add a record to a collection if it is not present, get all records,
etc...

I had difficulty finding a pattern to reuse formulaic code (Detail
here: http://noisylines.blogspot.com/2010/09/first-days-of-nodejs-and-mongodb.html)

Basically all the sample code I've found depends on things like db and
collection being in lexical scope, I want to pull out intermediate
layers for accessing my database and collection. Check out the
duplication here to see what I'm trying to avoid:
http://github.com/scrawlings/DelphiMethod/blob/a25d23a9fd7339bfb9e5e79b6fb3d580b5b9ecca/server/admin_services.js

So I've modified the node mongodb driver in my app. My modification
pushes a context object around which allows me to carry data through
explicitly. (Sorry, I haven't got the whole git fork and reference
thing sorted out, still learning my way around it. Feel free to point
me to instructions on how to make my changes easier to review.) Check
out the use of callback_context to see what I've tried:
http://github.com/scrawlings/DelphiMethod/blob/master/lib/mongodb/collection.js

To use it I basically end up tucking things into an extra parameter
like context in this snippet. It's optional except that if breaks the
two argument Collection.find(callback, options), (which is questioned
by someone before me):

function get_collection(err, db, context)
{
context.db =
db
db.collection(collection_name, handle_open_collection,
context);
}

Hacking the driver doesn't seem a good solution. Can anyone point me
to examples of a better pattern without hacking on the driver?

To have a look at the sort of things I've managed, which is pretty
much good enough for my purposes, please look at my database utilities
layer: http://github.com/scrawlings/DelphiMethod/blob/master/server/database_utilities.js

That results in some simple URL services which can be seen here:
http://github.com/scrawlings/DelphiMethod/blob/master/server/admin_services.js

It lets me turn things like this (of which there are several in the
app):

var db = new Db('delphi_method', new Server(host, port, {}),
{native_parser:true});

db.open(function(err, db) {
db.collection('users', function(err, collection) {
collection.find(function(err, cursor) {
cursor.toArray(function(err, items) {
var users = items.map(function(el) { return
el.name; });
sys.puts('JSON.stringify(users): ' +
JSON.stringify(users));
db.close();
res.send(JSON.stringify(users), {'Content-
Type': 'application/json'});
});
});
});
});

into, avoiding duplication of everything else:

db_utils.matching_items_finder('users')
(db_utils.finish(item_sender(res)));


Many thanks,
Julian.

[rick....@gmail.com]

I tend to wrap things if I want to reduce complexity etc

Take a look at this:

http://github.com/listentorick/mongodataprovider/blob/master/dataprovider.js

It works for me so feel free to submodule it. Note that it is a work in progress....

[rick....@gmail.com]

Note that the error handling in this code is bobbins - I do intend to sort it out soon :)

[jbk]

Yeah, to me it seemed that because the test code doesn't give error
handling much attention then code derived from it doesn't either.

I imagine myself providing error handlers as callbacks when I get
round to it. Interfaces that insist you check conditions bother me, if
the check is the same every time then it should be inside the
interface not outside.

On Sep 26, 8:51 am, rick.wa...@gmail.com wrote:
> Note that the error handling in this code is bobbins - I do intend to sort
> it out soon :)
>
>
>
> On Sun, Sep 26, 2010 at 8:51 AM, <rick.wa...@gmail.com> wrote:
> > I tend to wrap things if I want to reduce complexity etc
>
> > Take a look at this:
>

> >http://github.com/listentorick/mongodataprovider/blob/master/dataprov...
>
> > <http://github.com/listentorick/mongodataprovider/blob/master/dataprov...>It

> > works for me so feel free to submodule it. Note that it is a work in
> > progress....
>

> > On Sat, Sep 25, 2010 at 12:21 PM, jbk <julian.b.kel...@gmail.com> wrote:
>
> >> I'm working on a web application using node-mongodb-native and
> >> express.js. There are a number of URLs that perform similar actions:
> >> add a record to a collection if it is not present, get all records,
> >> etc...
>
> >> I had difficulty finding a pattern to reuse formulaic code (Detail
> >> here:

> >>http://noisylines.blogspot.com/2010/09/first-days-of-nodejs-and-mongo...
> >> )

>
> >> Basically all the sample code I've found depends on things like db and
> >> collection being in lexical scope, I want to pull out intermediate
> >> layers for accessing my database and collection. Check out the
> >> duplication here to see what I'm trying to avoid:
>

> >>http://github.com/scrawlings/DelphiMethod/blob/a25d23a9fd7339bfb9e5e7...

>
> >> So I've modified the node mongodb driver in my app. My modification
> >> pushes a context object around which allows me to carry data through
> >> explicitly. (Sorry, I haven't got the whole git fork and reference
> >> thing sorted out, still learning my way around it. Feel free to point
> >> me to instructions on how to make my changes easier to review.) Check
> >> out the use of callback_context to see what I've tried:
>

> >>http://github.com/scrawlings/DelphiMethod/blob/master/lib/mongodb/col...

>
> >> To use it I basically end up tucking things into an extra parameter
> >> like context in this snippet. It's optional except that if breaks the
> >> two argument Collection.find(callback, options), (which is questioned
> >> by someone before me):
>
> >>    function get_collection(err, db, context)
> >> {
> >>        context.db =
> >> db
> >>        db.collection(collection_name, handle_open_collection,
> >> context);
> >>    }
>
> >> Hacking the driver doesn't seem a good solution. Can anyone point me
> >> to examples of a better pattern without hacking on the driver?
>
> >> To have a look at the sort of things I've managed, which is pretty
> >> much good enough for my purposes, please look at my database utilities
> >> layer:

> >>http://github.com/scrawlings/DelphiMethod/blob/master/server/database...

>
> >> That results in some simple URL services which can be seen here:
>

> >>http://github.com/scrawlings/DelphiMethod/blob/master/server/admin_se...

[jbk]

Taken as read that all this is work in progress, thanks for putting
your stuff out there!

Is the 'db' passed to the callback in 'open' the same thing that
'open' was called on? Given you're not doing anything with that
callback I'm wondering if after you create your wrapper and you go
straight on to one of your operations, could you get end up trying one
of your operations before the database has been opened? I didn't look
closely at that part of the driver, but you may be saved by there
being a queue that keeps those things in order. If it's safe to do
then it's a nice simplifying move.

The difference is philosophical I guess, you're going down the path of
an OO wrapper, and I'll be following it with interest. Day by day I'd
promote it as the type of interface that will be easiest to work with.

I'm trying to turn it into more strictly functional thing. No
variables, no state sitting anywhere but on the stack, no depending on
nested scopes, just parameters, no fear of messing things up by making
wrong assumptions about state. But in my experience, it takes a bit of
time to get your head around the style, which is why I don't do stuff
like that in the office. It's just playing games with style. The
context object I've added makes that possible.

(Ideally, I'd like a callback object which encapsulated state and an
explicitly listed sequence of callbacks. Vaguely inspired by monads,
and the whole notion of avoiding implicit sequence. That would be a
new and very different driver, a project for another year when I've
got less on my plate.)

Thinking about how I'd use your library. Say I wanted to read
something from one record then process it and modify another record.
That's two dataproviders, and one has to be called from inside the
other's callback, or else you need some sort message passing or
blocking. Are you imagining that they will both belong to some
containing object that will hold the state from the earlier calls to
make it available to the later calls?

Thanks,
Julian.

On Sep 26, 8:51 am, rick.wa...@gmail.com wrote:

> I tend to wrap things if I want to reduce complexity etc
>
> Take a look at this:
>

> http://github.com/listentorick/mongodataprovider/blob/master/dataprov...
>
> <http://github.com/listentorick/mongodataprovider/blob/master/dataprov...>It

> works for me so feel free to submodule it. Note that it is a work in
> progress....
>
>
>
> On Sat, Sep 25, 2010 at 12:21 PM, jbk <julian.b.kel...@gmail.com> wrote:
> > I'm working on a web application using node-mongodb-native and
> > express.js. There are a number of URLs that perform similar actions:
> > add a record to a collection if it is not present, get all records,
> > etc...
>
> > I had difficulty finding a pattern to reuse formulaic code (Detail
> > here:

> >http://noisylines.blogspot.com/2010/09/first-days-of-nodejs-and-mongo...
> > )

>
> > Basically all the sample code I've found depends on things like db and
> > collection being in lexical scope, I want to pull out intermediate
> > layers for accessing my database and collection. Check out the
> > duplication here to see what I'm trying to avoid:
>

> >http://github.com/scrawlings/DelphiMethod/blob/a25d23a9fd7339bfb9e5e7...

>
> > So I've modified the node mongodb driver in my app. My modification
> > pushes a context object around which allows me to carry data through
> > explicitly. (Sorry, I haven't got the whole git fork and reference
> > thing sorted out, still learning my way around it. Feel free to point
> > me to instructions on how to make my changes easier to review.) Check
> > out the use of callback_context to see what I've tried:
>

> >http://github.com/scrawlings/DelphiMethod/blob/master/lib/mongodb/col...

>
> > To use it I basically end up tucking things into an extra parameter
> > like context in this snippet. It's optional except that if breaks the
> > two argument Collection.find(callback, options), (which is questioned
> > by someone before me):
>
> >    function get_collection(err, db, context)
> > {
> >        context.db =
> > db
> >        db.collection(collection_name, handle_open_collection,
> > context);
> >    }
>
> > Hacking the driver doesn't seem a good solution. Can anyone point me
> > to examples of a better pattern without hacking on the driver?
>
> > To have a look at the sort of things I've managed, which is pretty
> > much good enough for my purposes, please look at my database utilities
> > layer:

> >http://github.com/scrawlings/DelphiMethod/blob/master/server/database...

>
> > That results in some simple URL services which can be seen here:
>

> >http://github.com/scrawlings/DelphiMethod/blob/master/server/admin_se...

[rick....@gmail.com]

On Sun, Sep 26, 2010 at 10:36 AM, jbk <julian....@gmail.com> wrote:

Taken as read that all this is work in progress, thanks for putting
your stuff out there!

Is the 'db' passed to the callback in 'open' the same thing that
'open' was called on? Given you're not doing anything with that
callback I'm wondering if after you create your wrapper and you go
straight on to one of your operations, could you get end up trying one
of your operations before the database has been opened?

>> it could. I was planning on  removing the open from the constructor and updatinf methods such as findbyid so that they open the connection if it wasnt already open. 

>> when you say record do you mean collection? 

[jbk]

> > Thinking about how I'd use your library. Say I wanted to read
> > something from one record then process it and modify another record.
> > That's two dataproviders, and one has to be called from inside the
> > other's callback, or else you need some sort message passing or
> > blocking. Are you imagining that they will both belong to some
> > containing object that will hold the state from the earlier calls to
> > make it available to the later calls?
>
> when you say record do you mean collection?

I was thinking of documents within different collections.

Read some stuff from one collection and subsequently write to another
collection. Option 1, I write to the second collection in the callback
of reading from the first. Option 2, I stow the data somewhere
shareable and have a blocking construct to kick off the second step
when the first step signals that it's done.

The test code all seems in the style of option 1, getting data from
one callback to another via variables available to both through
lexical closure.

If I've cottoned on to the style you're aiming for, that data will
somehow be bound into an object that will be available to the
callbacks passed to your dataproviders. I'm not sure if you're
encouraging using lexical scope, or if you have and idea to use
objects in some way?

In my case I've hacked the driver so that I can push the data through
a context object that will be input to the callback.


Perhaps a basic demonstration app to test different wrappers and the
driver interface itself? The demo program must (1) read a doc from
collection A and (2) process that data in some way then (3) write the
resulting doc into collection B, the repeat (4) reading a doc from
collection C then (5) processing that doc and (6) writing into
collection D. The processing in steps (2) and (5) must be different
and the four collections must be different. How much code is required,
and how much duplication, to do this?


N.B. I fully accept that certain styles may be bulkier or require some
duplication but be conceptually easier. Depending on how we value
different code quality variables each of us may favour different
styles. I don't believe there will be one right way to do this. So as
we compare we should focus on what values we're trying to optimise,
(I've seen discussions about interfaces hijacked into time wasting
debates about values so I'm cautious).

[Olivier Tremblay]

How about Javascript's array notation?

From a pure javascript standpoint,

db.collection

is equivalent to

db['collection']

In Firebug:
>>> db = [];
[]
>>> db['collection'] = 'bob'
"bob"
>>> db.collection
"bob"

I'm no Mongo pro, but from my understanding it should work, meaning
you could easily extract a function and pass around a reference to the
collection as a var.

[Pau]

*shameless self-promotion*

I did a little wrapper for the mongo driver called mongolia.
http://github.com/masylum/mongolia

It basically allows you to attach javascript objects with collections
and then put all the logic related to that collection there.
It have also a little validation module and some helpers to work with
embedded objects.

Example:

var User = require('model').extend({
constructor: function (db) {
Model.call(this, db, 'users');
},
getCurrentUser: function (req, callback) {
this.mongoCall('findOne', {'_id': req.session.user._id},
callback);
}
});

var User = new (require('./models/user'))();
User.getCurrentUser(req, function (error, user) {
sys.puts('Welcome ' + user.name);
});

On Oct 15, 1:44 pm, Olivier Tremblay <oliviertrembla...@gmail.com>
wrote: