FleetDB and Concurrency?

[Judson Stephenson]

Hey Mark,

I began getting into my web app's api today and wanted to test the
limits of FleetDB + Nodejs, so I did:

> ab -n100 -c10 http://xxxxxx/users/jud

to have node execute this code:

db.open('http://127.0.0.1:3400');
var query = ["select", "users", {"where": ["=", "id",
req.params.user]}];
db.query(query, function(status, data){
res.send('jud');
});

db.close();

the problem is, that once the ab concurrency number gets any higher
than 2 or 3, FleetDB starts returning multiple results for the select
query, like:


[0,[{"id":"judson","email":"em...@a.com","pass":"$2a
$06$W6HIOcv4Q886cZ9/","active":true,"creation":
1296678178358,"api":false,"apiKey":false}]]
[0,[{"id":"judson","email":"em...@a.com","pass":"$2a
$06$W6HIOcv4Q886cZ9","active":true,"creation":
1296678178358,"api":false,"apiKey":false}]]

which causes invalid JSON errors. Then because the other pages are
waiting on their queries, which have somehow been pushed to the wrong
connection, they hang.

I am not sure if this is an issue with FleetDB or if new connections
to fleetdb are somehow shared in node (Unlikely, I think), or of
something is wrong with my implementation of node-fleet. I really
enjoyed FleetDB, but if there is a concurrency issue (in my code, or
maybe FleetDB) then it won't work :(

Does anything immediately jump out to you? Maybe I am overlooking
something because I have been at this a while ;)

thanks for your help,
Judosn

[Mark McGranaghan]

Hi Judson,

FleetDB should definitely be able to handle many concurrent
connections without error or performance degradation. The first thing
I'd be sure to check is that the client is reading the complete
response to its first query before it sends a second one (i.e. not
trying to pipeline requests). If the client is behaving correctly in
that respect (or you are only trying to issue one query per open/close
block, let me know and I will try to reproduce with a stripped down
version of your setup.

- Mark

[Judson Stephenson]

Thanks mark.

I am running express, which utilizes the connect server. And I have a
route setup /users/:id. It should be fairly trivial IMO. User connects
to server, server opens db, runs query, closes connection and send
`data` to the browser.

So the basic setup looks like:

var express = require('express'),
fleetdb = require('./inc-js/fleetdb.js'),
app = express.createServer();

/* open up fleetdb database */
var db = new fleetdb.Database();

app.use(express.bodyDecoder());
app.use(express.logger());

app.get('/users/:user', function(req, res){

db.open('http://127.0.0.1:3400');
var query = ["select", "users", {"where": ["=", "id",
req.params.user]}];
db.query(query, function(status, data){

db.close();
res.send(data);
});
});

app.listen(56796);
console.log('Express server started on port %s', app.address().port);


Each request to the page should open a new connection to fleetdb, so
I'm not sure why its getting the data from other queries.

Judson

On Feb 2, 4:44 pm, Mark McGranaghan <mmcgr...@gmail.com> wrote:
> Hi Judson,
>
> FleetDB should definitely be able to handle many concurrent
> connections without error or performance degradation. The first thing
> I'd be sure to check is that the client is reading the complete
> response to its first query before it sends a second one (i.e. not
> trying to pipeline requests). If the client is behaving correctly in
> that respect (or you are only trying to issue one query per open/close
> block, let me know and I will try to reproduce with a stripped down
> version of your setup.
>
> - Mark
>
> On Wed, Feb 2, 2011 at 2:31 PM, Judson Stephenson
>
>
>
> <J...@trollingtowers.com> wrote:
> > Hey Mark,
>
> > I began getting into my web app's api today and wanted to test the
> > limits of FleetDB + Nodejs, so I did:
>

> >> ab -n100 -c10http://xxxxxx/users/jud

[Mark McGranaghan]

As a quick sanity check, could you try moving

var db = new fleetdb.Database();

into the app.get('/...') block, right above db.open(...)? It seems
like otherwise all concurrent connections would be sharing the same db
object, though I'm not familiar enough with the implementation to know
for what that means internally.

- Mark

[Judson Stephenson]

Thanks for the extra set of eyes matt, In the end, I just moved the
db.open() to the beginning of the app and leave the db connection open
so there isn't the overhead of opening a new connection on every
request.

Thanks,
Jud