Holing a connection open for long

[Manu J]

1. I'm trying to hold a connection open for 30 seconds for long polling. Is this the right way to do it ? https://gist.github.com/2476212 (Note: Holding it open for only 5 seconds in this gist).

2.  Which redis driver should I use with goliath?  em-synchrony recommends the official redis-rb driver which has synchrony built in but  in https://github.com/jwarchol/redis_pubsub (Goliath with redis pub-sub), the author recommends using EM hiredis driver.

[Ilya Grigorik]

On Mon, Apr 23, 2012 at 11:38 PM, Manu J <manu...@gmail.com> wrote:

1. I'm trying to hold a connection open for 30 seconds for long polling. Is this the right way to do it ? https://gist.github.com/2476212 (Note: Holding it open for only 5 seconds in this gist).

Nope. That will close the connection immediately. See this example: https://github.com/postrank-labs/goliath/blob/master/examples/content_stream.rb#L37

After you indicate that this will be a streaming response you then have to explicitly call stream_send and feed it the output.

 

2.  Which redis driver should I use with goliath?  em-synchrony recommends the official redis-rb driver which has synchrony built in but  in https://github.com/jwarchol/redis_pubsub (Goliath with redis pub-sub), the author recommends using EM hiredis driver.

If you're streaming, you can use redis_pubsub just fine.. Look at the example above. You likely just want to connect the Redis callback to the stream callback and let them pipe the data automatically (also look at the amqp example in the repo.. it does exactly this).

ig

[Manu J]

Nope. That will close the connection immediately. See this example: https://github.com/postrank-labs/goliath/blob/master/examples/content_stream.rb#L37

That is intentional. The client cannot consume a streaming response but can hold a connection open for long waiting for a response. In that scenario,

is this the right pattern to use? 

 

2.  Which redis driver should I use with goliath?  em-synchrony recommends the official redis-rb driver which has synchrony built in but  in https://github.com/jwarchol/redis_pubsub (Goliath with redis pub-sub), the author recommends using EM hiredis driver.

If you're streaming, you can use redis_pubsub just fine.. Look at the example above. You likely just want to connect the Redis callback to the stream callback and let them pipe the data automatically (also look at the amqp example in the repo.. it does exactly this).

There has been some confusion :) What I wanted to know was which redis driver to use. 'redis-rb' with in built synchrony or 'em-hiredis'. em-synchrony website recommends the former but the author of redis pubsub example recommends the latter and recommended against using 'em-synchrony/hiredis'. from the pubsub readme,

" Be sure you don't use em-synchrony's hiredis driver in the config file. The execution is on the root fiber there and em-synchrony will not like that. Using the regular EM hiredis driver lets that work happen on the reactor loop without causing any problems"

I was not sure whether this affects redis-rb with built in synchrony also. 

[Manu J]

I think the confusion arose because I mentioned redis pubsub. I'm not using redis pubsub (yet) but I'm using redis and I want to know what's the right driver.

--

Manu

[Ilya Grigorik]

On Tue, Apr 24, 2012 at 11:26 PM, Manu J <manu...@gmail.com> wrote:

I think the confusion arose because I mentioned redis pubsub. I'm not using redis pubsub (yet) but I'm using redis and I want to know what's the right driver.

Both should be fine. I think more people have been using the main redis driver, so try that first.

ig

[Manu J]

OK. Sorry to bug you more but can you tell me if this is the right way to do what I wanted to do - https://gist.github.com/2476212 

Context: 

* Client will open a http connection and hold it open for long waiting for the server to respond. 

* Client cannot consume streaming responses

* Server checks every second whether data is available or if 5 seconds are up. If either of this happens, a response is send to the client

indicating whether data is present.

This seems like a naive solution but since the client cannot consume streaming responses, I could not think of another approach. I just don't want to

do something stupid :)

[Eric Marden]

Why bother holding the connection open if the client is just going to poll periodically anyway?


--
Eric Marden
http://ericmarden.com

[Manu J]

On Wed, May 2, 2012 at 3:30 AM, Eric Marden <eric....@gmail.com> wrote:

Why bother holding the connection open if the client is just going to poll periodically anyway?

Allows you to push data to client as soon as data comes in. Client immediately makes a new connection after it receives the data. 

Client doesn't "poll" in the strict sense. It just has one open connection to server at all time.

AFAIK this is a common technique - http://en.wikipedia.org/wiki/Comet_(programming)#Ajax_with_long_polling

Since the client doesn't stream there is a chance that intermediaries like proxies can close the connection after a certain time. 

Hence I don't keep the connection open indefinitely and close it at server side after a certain time.

[Ilya Grigorik]

Instead of use long-polling, you're probably much better off using SSE: http://www.igvita.com/2011/08/26/server-sent-event-notifications-with-html5/

There are good polyfills for SSE on older browsers as well (check comments on the post).

ig

[Manu J]

On Wed, May 2, 2012 at 10:49 AM, Ilya Grigorik <igri...@gmail.com> wrote:

Instead of use long-polling, you're probably much better off using SSE: http://www.igvita.com/2011/08/26/server-sent-event-notifications-with-html5/

There are good polyfills for SSE on older browsers as well (check comments on the post).

Thanks. I think I will use that. 

I'm curious though. What is the recommended way to handle Ajax long polling in Goliath?

--

Manu  

[Ilya Grigorik]

Thanks. I think I will use that. 

I'm curious though. What is the recommended way to handle Ajax long polling in Goliath?

Return "streaming response" constant in response_env, and then hook up some event emitter to your env.stream_send function, in which you can also call connection close. Pretty simple. :-)

ig

[Manu J]

Like this? https://gist.github.com/9a2d05804eb6a321ad26  

What are the advantages over just looping over with a EM::Synchrony sleep inside the loop ? One advantage is that it is easy to keep the connection alive by emitting a response frequently. 

--

Manu

[Ilya Grigorik]

I'm curious though. What is the recommended way to handle Ajax long polling in Goliath?

Return "streaming response" constant in response_env, and then hook up some event emitter to your env.stream_send function, in which you can also call connection close. Pretty simple. :-)

Like this? https://gist.github.com/9a2d05804eb6a321ad26  

What are the advantages over just looping over with a EM::Synchrony sleep inside the loop ? One advantage is that it is easy to keep the connection alive by emitting a response frequently. 

Yes, that's pretty close, but a few improvements:

- You need to cancel the timer in your on_close callback, otherwise the client can close the connection but the timer will continue firing

- If possible, instead of doing polling, use a pub-sub system to get notifications of updates -- although this is obviously dependent on the semantics of the service you're checking.

In theory, yes you can use a loop { EM::Synchrony.sleep n }, but you'll be better off using the timers - it's more idiomatic evented code.

ig

[Manu J]

Yes, that's pretty close, but a few improvements:

- You need to cancel the timer in your on_close callback, otherwise the client can close the connection but the timer will continue firing

- If possible, instead of doing polling, use a pub-sub system to get notifications of updates -- although this is obviously dependent on the semantics of the service you're checking.

In theory, yes you can use a loop { EM::Synchrony.sleep n }, but you'll be better off using the timers - it's more idiomatic evented code.

ig

Thanks for your patience Ilya :)  I would like to use redis pub sub but I haven't yet figured out how to use multiple channels yet. All the examples show

redis subscribing to a single channel and publishing to a single  EM::Channel.

[Ilya Grigorik]

Thanks for your patience Ilya :)  I would like to use redis pub sub but I haven't yet figured out how to use multiple channels yet. All the examples show

redis subscribing to a single channel and publishing to a single  EM::Channel.

Create a channel subscription to redis when the connection opens, and
within the same callback wire up the new message callback to the stream
send function. Yes, this does mean that each client will use its own
channel from Redis. However, if each channel is different to begin with
then you have no choice. If you do have shared channels, then put them in
config and share them between all subscribers via EM::Channel.

ig

[Manu J]

Splendid.  

My need is somewhere in between.  Closest analogy is chat with multiple rooms with users able to create new rooms. 

So I can't push the channel subscription into config. The technique outlined above is  perfect.  

Obviously this will work even for single channel setups. One reason to use EM::Channel in a single channel  setup is to 

reduce the number of connections opened to redis right? Are there any other?

--

Manu

[Ilya Grigorik]

On Thu, May 3, 2012 at 12:50 AM, Manu J <manu...@gmail.com> wrote:

One reason to use EM::Channel in a single channel  setup is to 

reduce the number of connections opened to redis right? Are there any other?

Correct. Also less data for Redis to push, etc. Fanout > point to point.. if you're distributing the same data.

ig