Re: channel api not working on production

[Kristopher Giesing]

I've seen problems with the Channel API where both sides of the connection were "half open", i.e. each side believed there was a connection but messages would not be sent.

I eventually replaced the Channel API with a node.js service running WebSockets.  The GAE app posts messages to this service using URLFetch.  It has worked really well so far, with only one outage incident (which was my own fault) and none of the issues with intermittent connectivity that I was seeing with Channel.

- Kris

On Sunday, December 30, 2012 12:00:42 PM UTC-8, 0914...@umt.edu.pk wrote:

I've recently faced the problem. I've implemented some collaborative work on my app. I've used the Channel API and it works really fine on localhost. After deploy, the "update" in the other client is never received. why?

[Johan Euphrosine]

Hi,

Do you still have this issue?

Are you sending and receiving messages from the same application version?

On Sun, Dec 30, 2012 at 12:00 PM, <0914...@umt.edu.pk> wrote:

I've recently faced the problem. I've implemented some collaborative work on my app. I've used the Channel API and it works really fine on localhost. After deploy, the "update" in the other client is never received. why?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cUYy7p9RIpgJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Aleksei Rovenski]

I can confirm this issue. It is there from August, it was the first time I noticed at least. For us it looks like this - everything connected and works, messages flow, then all of a sudden w/o any pattern in time or any other conditions, client simply stops receiving any messages. No callbacks, nothing. Then after an unspecified time messages may start going through again or may not. Sometimes it doesn't happen for a long time, sometimes it happens very often. We had to migrate to Pusher mostly to mitigate this issue. Channels are still used by us, but simply because now they are not affecting us so much and we postponed the effort to migrate fully in favor of other things. Anyway sooner or later we will do it, when smt new and nasty comes up :( I want to leave Channels as plan B if Pusher fails if this is not too huge effort to maintain..

PS. I fully understand that there is no way you can reproduce this issue with simple steps, so there can not be any fix. In order to catch the problem, you actually have to use the Channels extensively, because it requires time and scale. And I have a feeling nobody in GAE is using it, otherwise it wouldn't be so broken... sorry if the feedback sounds bit negative, but we lost time...

четверг, 3 января 2013 г., 0:43:12 UTC+2 пользователь Johan Euphrosine (Google) написал:

[alex]

http://code.google.com/p/googleappengine/issues/detail?id=4940

> https://groups.google.com/d/msg/google-appengine/-/IpGfS-GXdK8J.

[Aleksei Rovenski]

I see 400 SID errors, but it seems not to be critical, because we are able to recover easily, there is onerror callback called and client simply creates new channel, no big deal.. Much worse is when client and server do not know about the problem as I described above..

пятница, 4 января 2013 г., 10:20:20 UTC+2 пользователь alex написал:

[Aleksei Rovenski]

btw those are 2 issues I would like to be fixed, please star them:
Add timestamp to channel presence notifications
http://code.google.com/p/googleappengine/issues/detail?id=8151
Channel API batch send method
http://code.google.com/p/googleappengine/issues/detail?id=4506

:D

пятница, 4 января 2013 г., 10:31:56 UTC+2 пользователь Aleksei Rovenski написал:

[Kristopher Giesing]

This is exactly my experience.  I would recommend to anyone considering use of the Channel APIs that they find an alternative.

- Kris

[alex]

problem is onerror callback is not being always called so the client
isn't given a chance to recover. that turns out to be a big deal.

On Fri, Jan 4, 2013 at 9:31 AM, Aleksei Rovenski

> https://groups.google.com/d/msg/google-appengine/-/mE2dX3VlD5gJ.

[Glen Whitaker]

We are currently facing a similar problem where the client stops receiving messages from the server. What we have observed is the application will perform as expected, receiving messages opening up the channel on the initial few visits, however after a period of time the channel on the client will not receive any more messages despite being established.

Diagnosis has been performed on both local dev mode and production instances and behaviour is the same, except on production it takes a longer period of time ( approximately 1 hr ) before reaching this channel state of not receiving on the client. The problem has been observed in both the latest versions of Chrome and Firefox. 

Through use of lsof and netstat it was found that a large number of sockets were being held open by the browser in a CLOSE_WAIT state, which once the process (firefox) that was holding these sockets was closed and released the channel started to receive again, although at a very slow rate ( the slow rate on recovery could be caused by the current task queue latency ).

The problem does not exist if setting up a new channel with a new token every time, but when refreshing (disconnect/reconnect) the channel using the same token results in this problem where client stops receiving after a period of time. 

Still trying to find a solution to this problem.

[Vinny P]

Hello Glen,

As you can see from this lengthy discussion you've dug up, the Channel API can be quite temperamental, mildly speaking. I'm sorry to tell you that there is really no magic bullet solution, that's just the way that Channel API is right now.

However, here are a few solutions you might find more reliable:

1. Go back to standard AJAX/JSONP communications. Reliable, but can be taxing on the server.

2. Set up a node.js external server, set up a standard websockets (there are many libraries for this, one of the most popular is http://socket.io/#how-to-use ). The downside is that you'll have to manage and provision an external server.

3. Use an external hosted web sockets service, such as Firebase https://www.firebase.com/tutorial/#gettingstarted  Can get expensive quickly though, depending on your budget and use cases.

4. As you stated in your post, Channel API can get unstable if held open for long periods of time, so just force the opening of a new sockets periodically (you can do so by setting the duration_minutes argument in create_channel() function to a low number. Play around, see what works for you). Probably the best option.

Here is another discussion on the topic you might find useful: https://groups.google.com/forum/#!msg/google-appengine/9wMnC7DAa80/CehgVFpFubYJ 

I will note that the company I work for uses all of these methods (in different apps, of course!) in production applications, and they work quite well for us. 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

@GOV on AppDotNet: https://alpha.app.net/gov