Long Polling Service with Redis PubSub

[George Blazer]

Hi there,

I'm building a long polling service with Redis pubsub with one channel per user. This thing should scale up to 1 million concurrent idle connections. Is it crazy or it should work?

I have read this https://groups.google.com/forum/#!topic/goliath-io/YFohhoaOmyk and it's very similar, however I have few questions.

I haven't quite wrapped my head around EM timers for EM.run. Should I use timers for long polling or .run method?

Also I can't seem to get the response streaming properly.

https://gist.github.com/webdev/6606070

Deployment wise: I'm planning to use Nginx to reverse proxy to Goliath, and put Nginx's behind AWS ELB. That should work right? 

There is nothing weird about Goliath like i.e. Websockets and ELB right?

And that last but not least, what's a good way to benchmark and perf test Goliath.

ab, httperf, any other suggestions?

Thanks.

[George Blazer]

One more thing. I should be using em-redis to make sure Redis interaction doesn't block. Is that correct?

[George Blazer]

I can't quite wrap my head around blocking directives.

Here is a gist https://gist.github.com/webdev/6626171.

If I add a loop  EM.add_periodic_timer(1) { env.stream_send("\n") } that just flushes to the response then my main loop emits and flushes to the response as well.

The moment I remove add_periodic_timer I stop getting my data in the response.

Please help.

On Wednesday, September 18, 2013 1:08:15 AM UTC-7, George Blazer wrote:

[Ilya Grigorik]

On Thu, Sep 19, 2013 at 12:45 PM, George Blazer <gbl...@gmail.com> wrote:

I can't quite wrap my head around blocking directives.

Here is a gist https://gist.github.com/webdev/6626171.

If I add a loop  EM.add_periodic_timer(1) { env.stream_send("\n") } that just flushes to the response then my main loop emits and flushes to the response as well.

The moment I remove add_periodic_timer I stop getting my data in the response.

Please help.

What are you testing this with? Curl? If so, it's probably buffering your response. 

 

I'm building a long polling service with Redis pubsub with one channel per user. This thing should scale up to 1 million concurrent idle connections. Is it crazy or it should work?

I have read this https://groups.google.com/forum/#!topic/goliath-io/YFohhoaOmyk and it's very similar, however I have few questions.

I haven't quite wrapped my head around EM timers for EM.run. Should I use timers for long polling or .run method?

You don't need EM.run inside response, since you're already running within the reactor. EM.run is a noop in your example.

 

Deployment wise: I'm planning to use Nginx to reverse proxy to Goliath, and put Nginx's behind AWS ELB. That should work right? 

Sure, just make sure timeouts are configured properly all along the way (not sure what those are for ELB).

 

There is nothing weird about Goliath like i.e. Websockets and ELB right?

AFAIK, ELB does not support HTTP Upgrade for WebSocket, which means you would have to use it in "TCP mode".

 

And that last but not least, what's a good way to benchmark and perf test Goliath. 

ab, httperf, any other suggestions?

For testing, there are built-in helper modules for rspec -- check the wiki/committed specs and examples. For benchmarking, ab/httperf are good tools.

 

ig

[George Blazer]

Thanks Ilya.

In terms of buffering my response and using CURL for testing.

I'd like to be able to use CURL during testing. Any way I can have CURL not buffer the response?

[George Blazer]

I tried curl -N (no buffering), but still not getting anything omitted in the response.

[Ilya Grigorik]

Have you tried running the test example committed in the repo? We know that works.. make sure your tools work with it.

--
You received this message because you are subscribed to the Google Groups "Goliath.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goliath-io+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.