node and database connection pool

[Reza Razavipour]

If node is single threaded and I can only be making one database call at any time, what is the point of having a pool of db connnections?

[Angel Java Lopez]

JavaScript in Node is single thread. But async I/O is derived to LibUv, that uses threads. Usually, a DB driver calls the database server, using sockets. Then, that communication will be handle by LibUv, returning the results in a callback. Your single JS threads is available for other tasks

Angel "Java" Lopez

@ajlopez

On Wed, Feb 26, 2014 at 2:14 PM, Reza Razavipour <reza.ra...@gmail.com> wrote:

If node is single threaded and I can only be making one database call at any time, what is the point of having a pool of db connnections?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Reza Razavipour]

makes perfect sense, thank you

Regards,
Reza

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/-_pSIeB7Vws/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Matt]

But unfortunately isn't how it works at all, unless the database driver in question is using C libraries which use the thread pool (which most of them don't these days).

It's all about the event loop. Here's how an event loop works: http://baudehlo.com/2013/02/14/how-an-event-loop-works/

[Angel Java Lopez]

Ummm... You can write a driver in JS code that use require('net'). And in that module, the socket I/O is, at the end, managed by LibUv. The same for require('fs') for file I/O.

Am I right?

And yes, the driver could be implemented in C, and then, it should call LibUv in some way.

Angel "Java" Lopez

@ajlopez

[Angel Java Lopez]

Some more concrete example, to support my assertion, and to have a clear picture of what can be used.

I just found:

https://github.com/felixge/node-mysql

It manage a pool:

https://github.com/felixge/node-mysql/blob/master/lib/Pool.js

A pool manage a list of available Connection

https://github.com/felixge/node-mysql/blob/master/lib/Connection.js

and connection use require('net')

That's the magic! ;-) The driver author uses LibUv indirectly.

Angel "Java" Lopez

@ajlopez

[Reza Razavipour]

Matt,

just so I understand, you are saying if a node package such as mongodb provides connection pooling, they have to be implementing the package using a pooling system that is implemented in C?

[Andrey]

Some databases (MySQL for example) don't allow you to send next query before receiving result of a previous one.

The only way to have queries run in parallel is have each query in separate connection. 

Even when query is very short it's a good idea to use connection pool to overcome network latency to DB server. 

[Matt]

On Wed, Feb 26, 2014 at 2:45 PM, Angel Java Lopez <ajlop...@gmail.com> wrote:

Ummm... You can write a driver in JS code that use require('net'). And in that module, the socket I/O is, at the end, managed by LibUv. The same for require('fs') for file I/O.

Am I right?

And yes, the driver could be implemented in C, and then, it should call LibUv in some way.

Libuv has only one thread for the event loop.

It does provide threads for things that don't work with the event loop (like a non-async database library, or filesystem operations), but it's mostly irrelevant here.

[Matt]

On Wed, Feb 26, 2014 at 2:50 PM, Angel Java Lopez <ajlop...@gmail.com> wrote:

I just found:

https://github.com/felixge/node-mysql

It manage a pool:

https://github.com/felixge/node-mysql/blob/master/lib/Pool.js

A pool manage a list of available Connection

https://github.com/felixge/node-mysql/blob/master/lib/Connection.js

and connection use require('net')

That's the magic! ;-) The driver author uses LibUv indirectly.

And that emphatically does NOT use threads in this case.

[Matt]

On Wed, Feb 26, 2014 at 4:15 PM, Reza Razavipour <reza.ra...@gmail.com> wrote:

Matt,

just so I understand, you are saying if a node package such as mongodb provides connection pooling, they have to be implementing the package using a pooling system that is implemented in C?

No I'm saying nothing like that at all.

[Angel Java Lopez]

Regarding:

That's the magic! ;-) The driver author uses LibUv indirectly.

And that emphatically does NOT use threads in this case.

Yes, you are right, so I said "uses LibUv indirectly", without adding "with threads", now.

Only for complete the topic, I also mentioned require('fs'). I'm not sure the scope of the use of threads in filesystem, but AFAIK (in Windows?) lib uv could use a pool of thread for some operations that are not easily mapped to async IO in host operating system

I don't know the current state of the art, but:

http://stackoverflow.com/questions/20032078/libuv-uses-blocking-file-system-calls-internally-why-how

Back to original question, all the magic goes to libuv, without the need of blocking js thread. And it was mentioned the need of many sockets (then connection pool) in case the server doesn't allow the use of the same one for multiple operations.

Angel "Java" Lopez

@ajlopez

--

[Matt]

On Thu, Feb 27, 2014 at 10:31 AM, Angel Java Lopez <ajlop...@gmail.com> wrote:

Only for complete the topic, I also mentioned require('fs'). I'm not sure the scope of the use of threads in filesystem, but AFAIK (in Windows?) lib uv could use a pool of thread for some operations that are not easily mapped to async IO in host operating system

Yes, the thread pool is used for fs operations.