Mongo db connection reuse

[Chris Scribner]

Hi,

I'm wondering what the best pattern for connection reuse is with the
native driver for a normal web site.

1. Open a connection for each database operation (assuming some
connection pooling on the backend)
2. Open a connection for each web request, use it for all operations
within a request, then close it at the end of the request
3. Any others?

I'm also wondering if there's any issues with making database calls
using the same connection as a previous database call, before the
previous call has returned. If there are issues with this, then #2
above wouldn't work well. If it does work, then might it be possible
to share a single connection (or randomly pick from a pool) for all
clients?

Thanks,

Chris

[christkv]

On Jul 14, 7:45 pm, Chris Scribner <scr...@gmail.com> wrote:
> Hi,
>
> I'm wondering what the best pattern for connection reuse is with the
> native driver for a normal web site.
>
> 1. Open a connection for each database operation (assuming some
> connection pooling on the backend)

Don't

> 2. Open a connection for each web request, use it for all operations
> within a request, then close it at the end of the request

Don't either

> 3. Any others?

Open a connection before you start the app and reuse it.

var db = new Db(....)
db.open(function(err, db) {
// Start your app and use the db object
})

Don't worry about pooling unless you can prove you need it to
yourself. Scale the number of node instances first before setting a
poolsize on you connection larger than 1. A core i7 processor can
easily handle 8 node processes.

If you do need pooling here is an example Server object

new Server("127.0.0.1", 27017, {auto_reconnect: true, poolSize: 4}))

[Samyak Bhuta]

Hi,

Since which version pooling is available ? I am using v 0.9.4 , will
it work with it ?

Regards,

Samyak

[christkv]

newest version, pass poolSize:2 or something to the Server class. Have
a look at the integration test for usage. Remember that things will
execute out of order so you're code might break in unexpected places.

[Samyak Bhuta]

Hi Crist,

You are superb with the prompt reply, always, thanks you for that !!

I am creating single db instance where autoreconnect is set to true as
well as strict mode is also set to true. I am using single db object
through out the life cycle of application. Here is the relevant code
... https://gist.github.com/1092746

Some time and only some time, I am getting an error during
getCollection method, saying collection someCollectionName doesn't
exist.

e.g.
{ stack: [Getter/Setter],
arguments: undefined,
type: undefined,
message: 'Collection items does not exist. Currently in strict mode.' }

When I encounter this error the collection indeed exists. This error
is not something which I get all the time. Sometime it is a successful
getCollection operation but sometime it is with this error. When this
strange error occurs, I am also putting output of db object, to see
the status of the connection using db.state, and it is showing to be
'connected' only.

Also, sometime the execution of node itself will get stuck up on this
very method. That is to say any of the successful scenario or the
erroneous/exceptional scenario will not be executed. It just freezes.

Further, I tested it with v 0.9.4 with no connection pulling and with
latest v 0.9.6-7 and connection pooling set to poolSize: 2, as
recommended.

What could be wrong in my code ?

Regards,

Samyak

[christkv]

That error indicates that the collection does not exist. So looks like
you have some sort of runtime condition where the collection has not
been created before you try to operate on it.

My recommendation

Don't run the driver in strict mode and use safe when doing inserts,
update, remove instead. This way you don't need to explicitly create a
collection before using it. And you can manage when you really need
feedback from mongodb. strict/safe fires of 2 messages to the db so
it's costly. That's the way mongodb works as insert/update/remove are
async operations and you need to ask for the error to check the result
of the operation.

so

insert(somedoc, {safe:true}, function(err, result) {})
update(somedoc, {safe:true}, function(err, result) {})
remove(somedoc, {safe:true}, function(err, result) {})

Best of luck

On Jul 19, 5:22 pm, Samyak Bhuta <samyak.bh...@gmail.com> wrote:
> Hi Crist,
>
> You are superb with the prompt reply, always, thanks you for that !!
>
> I am creating single db instance where autoreconnect is set to true as
> well as strict mode is also set to true. I am using single db object
> through out the life cycle of application.  Here is the relevant code

> ...https://gist.github.com/1092746

[Samyak Bhuta]

The error is coming while trying to avail collection using
db.collection(...), so how could a collection have created before this
method, we run this method to create collection, don't we ?

[christkv]

in strict mode you have to specifically create the collection.
collection() only checks if there is a collection available and fails
if it's not in strict mode.

I suggest checking that collections exists when you boot up and
creating any missing ones or use safe instead.

[Samyak Bhuta]

Oh I see, but the db.collection(...) method is used many times to
avail the same collection (during the life cycle of db connection !!),
and the failure is frequent, not always, for same code. Infact, I am
struggling to replicate it at my will making debugging harder !!! How
could be it be possible that the collection which I just availed for
scenario A/functionality A is not not available for scenario
B/functionality B ?

[Samyak Bhuta]

A quick update, when I tried connecting to some mongo server on cloud
( I tested for both monghq and mongolab) this error is coming up (
again not always but frequently ) while If I run the mongo instance
locally and run the scenario the error is not coming .. does this help
to understand what is the issue at the core ?

[christkv]

Nope, please fork the driver project, replicate the problem with an
integration test and send me a pull request. Sorry that's pretty much
all I can do.

On Jul 19, 7:44 pm, Samyak Bhuta <samyak.bh...@gmail.com> wrote:
> A quick update, when I tried connecting to some mongo server on cloud
> ( I tested for both monghq and mongolab) this error is coming up (
> again not always but frequently ) while If I run the mongo instance
> locally and run the scenario the error is not coming .. does this help
> to understand what is the issue at the core ?
>
>
>
>
>
>
>
> On Tue, Jul 19, 2011 at 4:51 PM, Samyak Bhuta <samyak.bh...@gmail.com> wrote:
> > Oh I see, but the db.collection(...) method is used many times to
> > avail the same collection (during the life cycle of db connection !!),
> > and the failure is frequent, not always, for same code. Infact, I am
> > struggling to replicate it at my will making debugging harder !!! How
> > could be it be possible that the collection which I just availed for
> > scenario A/functionality A is not not available for scenario
> > B/functionality B ?
>