Queue requests

[Alan Hoffmeister]

Hello fellows,

I want to queue my POST requests, someone known a link where I can read more about requests, Node.js and queue manager (my preference is RabbitMQ).

Thanks!

[Martin Wawrusch]

No docs that I know of, but take a look at the amqp npm module, it works fine with RabbitMQ (at least it used to a year back when I worked with it).

Other then that it is simply a matter of defining a message format, pushing that onto the queue, and have some separate process(es) working the queue.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Alan Hoffmeister]

The major problem is the logic, I had mastered  amqp already.

[Angel Java Lopez]

Just curious... 

Why you need to queue the POST request?

An in-memory queue is enough? Or you need to process the POST request in another process? (I can't imagine why/how)

[Martin Wawrusch]

Here are some docs, just a piece of advice - this can easily result in overkill, only use when it is really necessary:

http://www.soapatterns.org/synchronous_response_deferred_processing.php

http://martinfowler.com/bliki/CQRS.html

Also, it might be feasible to check out iron.io - iron worker product, which works with node.js too. It basically takes care of message queues, worker processes etc and is super fast (< 1s response time)

@Angel

You do this to prevent clogging of the queue on a single machine, and to easier distribute load, and to recover in case of error scenarios.

[Angel Java Lopez]

Thanks! Hmm... interesting... but I missed some part, Martin. If I understand your idea, the consumer(s) of the queue is in another machine. That's is Ok, it compiles in my neuron ;-) My problem is about "POST request".

The initial POST request reach Machine A, from Client 1. At the end, it's a socket connection, that http module gives to us as req/res. How to attend that incoming request in Machine B? The client (maybe a browser) knows Machine A, the open socket channel knows Machine A, not B, and it already started to send POST data to it, maybe a file upload, I guess. The only solution I can see: open a socket from A to B, and pipe to it the incoming request (I apologize... my English, I'm struggling trying to express the idea ;-). 

But I could misunderstand the initial question..

[Martin Wawrusch]

There are a couple ways to do this, for example:

POST request with payload -> Machine A

within request, machine A takes the payload and adds a message with it to the queue,

then returns with some status message, perhaps containing a reference to the task entry.

Machine B, upon pulling the message from the queue, processes the message, updates the database(s), updates caching layers (this is the tricky part), marks the message as processed, and perhaps sends a push notification to the originator of the POST request, probably through a common gateway, or the client just polls.

For long file uploads, if you have sticky load balancing you just store the file chunks locally, once assembled upload it to s3 or something, then create a queue message that references the s3 file.

If you don't have sticky load balancing, then you need to save the data chunks directly to s3, and keep some status info in redis or something.

Note that this is not just for large file uploads, in a lot of cases you do this for normal database requests as well. In addition to that it can get very complicated very fast if you have intermediate caching layers.

[Angel Java Lopez]

Martin, thanks for the detailed explanation. Now, it's clear to me.

I see the light ;-) you are queuing the PAYLOAD, not the POST request, the request object and transfer that it was born with the post.

Now I understand the situation (I hope ;-) I see three main use cases:

a- you need to queue the payload, process it, without sending a response to the client.

b- you need to queue the payload, process it, and the client needs a response in the associated (to original request) response.

c- you need to queue the payload, process it, and the client needs a result, in the future.

Case a: can be solved easily, only en-queue the data/message, and process it in a consumer process.

Case c: it's like case a, and, as you say, a client could poll the result. I used this case in a game demo past year: the user started a new game, send the request to main web site (not node yet), the web site send the payload to an Azure queue, Azure workers process the command, making the provisioning of the new game (board, initial position, maybe notify to other players), and the client (browser) polls a blob storage that has the game board/room status (ready, in progress.. etc).

Case b: hmmm.. I could make a callback function:

var newfn = function(result) {

     ///....

     res.write(result); // res is a free "outer" variable, in this closure

     res.end();

}

 save it in an object as

var nc = ncallbacks++;

callbacks[nc] = newfn;

and put the nc value in the message to put into the queue. Many consumers can process that messages, and then, sent a result to a second queue, with the nc value in the result. The original web site consumes that queue, and then, something like:

nc = message.nc;

result = message.result;

cb = callbacks[nc];

delete callback[nc];

cb(result);

If you have many web apps, there is a result queue by each one. The message sent to the first queue, should have not only nc, but the name of the queue associated to the inicial web app.

Maybe, instead of a result queue, a pub/sub-like solution could be used.

But I guess case b is the difficult one. 

Another alternative: instead of queue, use an RPC solution, like dnode. And the web app connects and balance the loads to many RPC servers.

I used the callback "old trick" in

https://github.com/ajlopez/SimpleRemote/blob/master/lib/simpleremote.js#L37