Channel API not working in production

[Praveen Kumar]

I have implement struts based chat application using channel API
trying to implement the chat application., The Sample struts chat
application works in local enviroinment .But in production
enviroinment totally it sucks.Please help

[Ikai Lan (Google)]

Praveen,

Try asking this question on StackOverflow:

https://groups.google.com/forum/?fromgroups#!topic/google-appengine/Z6XN_64cA7w

Also, it would be helpful to provide more details when you do. "It sucks" is terrible information. That kind of feedback is not actionable.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Pascal Patry]

Hi,

Channels on AppEngine are picky. There are multiple issues that I had to
work around for an application that is currently in production:
1. Channels aren't working on backends
2. No API is available to mobile
3. The implementation is different from the test and production server
4. Connection/Disconnection handlers aren't reliably called

I could go more in details on each of those issues and explain how I
worked around them, but if you are still evaluating possible solutions, I
would say, avoid them at all cost. At the end, I had to implement a
polling & memcache solution in order to *reliably* use channels. It did
cost us a lot of time and I wouldn't repeat this error again.

The fact is, it's the only solution that let you have a persistent
connection on AppEngine. Sadly, with all the efforts I've put in there,
I would like to re-use our current solution/model but it did prove
itself to be very costly in terms of time and material, so it won't
happen.

Because of this, I'm no longer developing any application on GAE.

That's my feedback but if you are looking for more tips & tricks about
those, let me know. I'm always happy to help.


On August 31, 2012 10:14:25 Miguel Alfonso Varela Fonseca wrote:
> Hi All.
>
> I've recently faced the same problem. I've implemented some collaborative
> work on my app. I've used the Channel API in dev mode and it works really
> fine. After deploy, I've tried to do the same actions as in development,
> but the "update" in the other client is never recieved. I've added some
> logs to my code and I find the lines next lines are reached.
>
> final ChannelMessage channelMessage = new ChannelMessage(clientId,
> myMessage);
> ChannelServiceFactory.getChannelService().sendMessage(channelMessage);
>
> After this, on the client side nothing happens.
>
> In the App Engine console, I don't find none "warning", "error", or even,
> "info" logs related to Channel API (Just logs from server I've added), So I
> really don't know the cause of the failure.
>
> I've got Channel API working on a two weeks deploy, the actual version
> works fine in dev mode, and the actual version has the same configuration
> of the Channel API, so, I think, the issue is no related to my code (I
> hope).
>
> I've read in some threads Channel API presented some production issues some
> time ago. How can I know if there is a new issue on the whole service?
>
> Please, this release is really important to my clients, so I hope really
> quick answares.
>
> Thank you so much.
>
> PD. in StackOverflow I've not found nothig related or usefull, even less in
> the
> https://groups.google.com/forum/?fromgroups#!topic/google-appengine/Z6XN_64cA7w therad.
> regardless of platform, the really important things are the answers.
>
>
>
> El miércoles, 14 de marzo de 2012 14:42:44 UTC-5, Ikai Lan (Google)
> escribió:

> >
> > Praveen,
> >
> > Try asking this question on StackOverflow:
> >
> >
> > https://groups.google.com/forum/?fromgroups#!topic/google-appengine/Z6XN_64cA7w
> >
> > Also, it would be helpful to provide more details when you do. "It sucks"
> > is terrible information. That kind of feedback is not actionable.
> >
> > --
> > Ikai Lan
> > Developer Programs Engineer, Google App Engine
> > plus.ikailan.com
> >
> >
> >

> > On Mon, Mar 12, 2012 at 3:58 AM, praveen <praveensu...@gmail.com<javascript:>

> > > wrote:
> >
> >> I have implement struts based chat application using channel API
> >> trying to implement the chat application., The Sample struts chat
> >> application works in local enviroinment .But in production
> >> enviroinment totally it sucks.Please help
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.

> >> To post to this group, send email to google-a...@googlegroups.com<javascript:>
> >> .

> >> To unsubscribe from this group, send email to

> >> google-appengi...@googlegroups.com <javascript:>.

[Kristopher Giesing]

On Friday, August 31, 2012 12:29:20 PM UTC-7, Pascal Patry wrote:

Hi,

Channels on AppEngine are picky. There are multiple issues that I had to
work around for an application that is currently in production:
1. Channels aren't working on backends

Argh, seriously? That is a huge problem for me :(

 

2. No API is available to mobile

I worked around this by embedding the jsapi code in my application.  It seems to work fine... so far.

 

3. The implementation is different from the test and production server

Yes, that is a pain.  I had to add a wrapper class to my application to choose between embedded test and prod implementations dynamically.

 

4. Connection/Disconnection handlers aren't reliably called

Are you talking about this? If so it was fixed: http://code.google.com/p/googleappengine/issues/detail?id=7098

The fix isn't public yet but there's a patch in the thread.

- Kris

[Aleksei Rovenski]

Hi,

I have to agree with Pascal. Channel API has too many issues and though I have worked around most of them to a certain degree when it started to be good enough, I would not repeat this if I would be still planning my application. Some more issues that I can easily add to the list above:

- doesn't support batch sending (if you want to broadcast same message)

Note that the cost you see in the dashboard for Channel API is not the full cost. You also have to consider that your instances will spend some time sending the messages. Usually the API call to send a message takes 20-40 millis, but on occasions it can take a couple of seconds. And if you are broadcasting your instances may end up spending most of the time doing sendMessage calls..

There is an issue for that, please star it: http://code.google.com/p/googleappengine/issues/detail?id=4506

- 400 SID errors

Usually it doesn't happen too often, but sometimes you will get them right after your client connected and then again...

There is an issue for that, feel free to star it: http://code.google.com/p/googleappengine/issues/detail?id=4940

- Sometimes channels are just dead

You create a channel, there are no errors or anything, but no matter how many messages you send, it just won't receive any. The token is still valid, so your client can try to reconnect, but it won't help, it has to create a new channel with a new token. But how to detect this kind of issue until it is too late and user has left your app for good? The problem is sometimes channel takes 20+ seconds before it receives messages..

- Quota of 60 connects per minute (1 per sec)

If I succeed, it won't be enough for my app. I have no idea how responsive is google in increasing the quota..

+1 for presence notifications are extremely unreliable

To a degree that I had to duplicate them, my app sends its own callbacks. What is unreliable you may ask? Disconnect notification for example may come *minutes* after. So my client already handled the situation and connected to another channel long ago. If I would have waited for presence notification, I would have lost the user. Or even worse on some days the notification may not come at all. 

And because you can never be totally sure that you have not missed something or channel doesn't give you a new trick, I had to add shamefull lazy polling to make sure my users are still connected :(

PS. I'm trying out Pusher and it looks good. 

PSS. I still like Appengine, but Channel API is really not ready for production apps.

суббота, 1 сентября 2012 г., 7:32:56 UTC+3 пользователь Kristopher Giesing написал:

[vlad]

+1  Channels is probably the most overhyped but underwhelming feature on GAE. It was launched with lots of hype and high expectations. Still after 2 years to is mostly unusable. GAE team is better off killing Channels as a feature to free up resources and focus on delivering websockets.

[Drake]

I think they know this, which is why there are “Endpoints” which aren’t the same thing but are described for the same use cases, and suck in their own special way.

 

 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/xxXmgwM8N0gJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Richard Watson]

Aleksei - please tell us how you go with Pusher.

[Aleksei Rovenski]

I don't know what is your use case, but for mine Pusher is better by definition because it seamlessly supports broadcasting.

So far I'm using it in parallel with google channels for not so important events and it looks good. It works through web sockets with fallback to flash, but the stats I gather show me that eventually more than 95% of users get connected (apprx the same as with google). They have pricing model that I like more, fixed price and only one quota (I'm ware of) - number of simultaneous connections, lets say 500 ($49), so no hidden costs here. I understood the quota is soft, they will ask you to upgrade if exceeded. Those connections can be grouped simply by subscribing to any number of channels and there is no restriction on number of connections to single channel. So all users can be subscribed to one global channel for common news, then to room specific channels for scoped messages and finally to personal (protected) ones for private messaging. API is very simple, java lib is available (simple wrapper for a single urlfetch call). It reconnects automatically if network is down and doesn't expire as google tokens. There are "web hooks" that can replace google's presence notifications. So the 3 reasons I would recommend it are:

1) effortless broadcasting

2) minimum dev effort (I beleive the time spent on channel api problems exceeds few years of pusher fee)

3) I bet they are much more responsive as they can not ignore their users as Google does

To be fair I don't like flash fallback of Pusher, so this has to be mentioned..

In general the fundamental problem of GAE IMHO is that it was rebranded as a premium service to justify the costs, which is fine by me as some of the features are fantastic, but it doesn't have a corresponding QoS in some of the areas and support is not on a level you would expect from a premium service.

PS. One final thing I forgot to mention about Channel API problems is that it seems some of the users simply can not load the channel javascript lib which is hosted by google... What the hell...

понедельник, 3 сентября 2012 г., 8:26:14 UTC+3 пользователь Richard Watson написал:

[Johan Euphrosine]

On Fri, Aug 31, 2012 at 9:28 PM, Pascal Patry <ppa...@spectrumdt.com> wrote:
> Hi,
>
> Channels on AppEngine are picky. There are multiple issues that I had to
> work around for an application that is currently in production:

Thanks for your feedback, please make sure you fill new
defects/feature requests or star existing ones about those issues so
you can be notified about our progress:
http://code.google.com/p/googleappengine/issues/list?cursor=googleappengine%3A7718&q=Component%3DChannel

> 1. Channels aren't working on backends

http://code.google.com/p/googleappengine/issues/detail?id=5123

> 2. No API is available to mobile

http://code.google.com/p/googleappengine/issues/detail?id=4226
http://code.google.com/p/googleappengine/issues/detail?id=4189

> 3. The implementation is different from the test and production server

Care to elaborate? (file new Defect if necessary)

> 4. Connection/Disconnection handlers aren't reliably called

Does this correspond to this?
http://code.google.com/p/googleappengine/issues/detail?id=5975

> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Johan Euphrosine]

On Sun, Sep 2, 2012 at 10:16 AM, Aleksei Rovenski
<aleksei....@gmail.com> wrote:
> Hi,
>
> I have to agree with Pascal. Channel API has too many issues and though I
> have worked around most of them to a certain degree when it started to be
> good enough, I would not repeat this if I would be still planning my
> application. Some more issues that I can easily add to the list above:

Hi Aleksei,

Thanks for your feedback,

>
> - doesn't support batch sending (if you want to broadcast same message)
> Note that the cost you see in the dashboard for Channel API is not the full
> cost. You also have to consider that your instances will spend some time
> sending the messages. Usually the API call to send a message takes 20-40
> millis, but on occasions it can take a couple of seconds. And if you are
> broadcasting your instances may end up spending most of the time doing
> sendMessage calls..
> There is an issue for that, please star it:
> http://code.google.com/p/googleappengine/issues/detail?id=4506

I just escalated this bug to the Channel engineering team.

>
> - 400 SID errors
> Usually it doesn't happen too often, but sometimes you will get them right
> after your client connected and then again...
> There is an issue for that, feel free to star it:
> http://code.google.com/p/googleappengine/issues/detail?id=4940

Just pinged the team about this.

>
> - Sometimes channels are just dead
> You create a channel, there are no errors or anything, but no matter how
> many messages you send, it just won't receive any. The token is still valid,
> so your client can try to reconnect, but it won't help, it has to create a
> new channel with a new token. But how to detect this kind of issue until it
> is too late and user has left your app for good? The problem is sometimes
> channel takes 20+ seconds before it receives messages..

Is it similar to
http://code.google.com/p/googleappengine/issues/detail?id=7239 ?
If yes, can you comment with more details about your application
(appids, logs, etc).

>
> - Quota of 60 connects per minute (1 per sec)
> If I succeed, it won't be enough for my app. I have no idea how responsive
> is google in increasing the quota..

Feel free to fill a production issue for your application.

>
> +1 for presence notifications are extremely unreliable
> To a degree that I had to duplicate them, my app sends its own callbacks.
> What is unreliable you may ask? Disconnect notification for example may come
> *minutes* after. So my client already handled the situation and connected to
> another channel long ago. If I would have waited for presence notification,
> I would have lost the user. Or even worse on some days the notification may
> not come at all.
> And because you can never be totally sure that you have not missed something
> or channel doesn't give you a new trick, I had to add shamefull lazy polling
> to make sure my users are still connected :(

Is it similar to
http://code.google.com/p/googleappengine/issues/detail?id=5975 or
something else?

>
> PS. I'm trying out Pusher and it looks good.
> PSS. I still like Appengine, but Channel API is really not ready for
> production apps.
>
> суббота, 1 сентября 2012 г., 7:32:56 UTC+3 пользователь Kristopher Giesing
> написал:
>>
>>
>>
>> On Friday, August 31, 2012 12:29:20 PM UTC-7, Pascal Patry wrote:
>>>
>>> Hi,
>>>
>>> Channels on AppEngine are picky. There are multiple issues that I had to
>>> work around for an application that is currently in production:
>>> 1. Channels aren't working on backends
>>
>>
>> Argh, seriously? That is a huge problem for me :(
>>
>>>
>>> 2. No API is available to mobile
>>
>>
>> I worked around this by embedding the jsapi code in my application. It
>> seems to work fine... so far.
>>
>>>
>>> 3. The implementation is different from the test and production server
>>
>>
>> Yes, that is a pain. I had to add a wrapper class to my application to
>> choose between embedded test and prod implementations dynamically.
>>
>>>
>>> 4. Connection/Disconnection handlers aren't reliably called
>>
>>
>> Are you talking about this? If so it was fixed:
>> http://code.google.com/p/googleappengine/issues/detail?id=7098
>>
>> The fix isn't public yet but there's a patch in the thread.
>>
>> - Kris
>>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/YPNuW4F8A0kJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Johan Euphrosine]

Are you refererring to:
http://code.google.com/p/googleappengine/issues/detail?id=4764

Or something else?

>
> понедельник, 3 сентября 2012 г., 8:26:14 UTC+3 пользователь Richard Watson
> написал:
>>
>> Aleksei - please tell us how you go with Pusher.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/xKoXd078q3IJ.

>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Aleksei Rovenski]

Hi!

 

I just escalated this bug to the Channel engineering team.

Very glad to hear this, thanks!

 

>
> - Sometimes channels are just dead
> You create a channel, there are no errors or anything, but no matter how
> many messages you send, it just won't receive any. The token is still valid,
> so your client can try to reconnect, but it won't help, it has to create a
> new channel with a new token. But how to detect this kind of issue until it
> is too late and user has left your app for good? The problem is sometimes
> channel takes 20+ seconds before it receives messages..


Is it similar to
http://code.google.com/p/googleappengine/issues/detail?id=7239 ?
If yes, can you comment with more details about your application
(appids, logs, etc).

Actually it is. I remember when I posted that bug it was a bad day for Channel API and it lasted for a day or two. I still see it, but only in the google analytics stats I gather, I'm not able to reproduce it myself, but it seems to happen to at least 1.5% of our users. When client is connected, there is a handshake message sent to it, if it doesn't receive it in 10 seconds, it is repeated and then again. If client doesn't receive any messages in 60 seconds, the channel is considered dead and new one is created (this event is then logged in analytics). Obviously many users are not waiting for so long, so 1.5% is an optimistic estimate. In such cases clients do not get onError() or onClose() callbacks, because those would trigger reconnection which doesn't happen. Both client and server see the channel as connected (client has socket opened and server got connected presence), sendMessage() is silent as well, so I can not really give you any logs. I understand that this doesn't allow you to reproduce this bug, I really wish I could give more precise information.

>
> +1 for presence notifications are extremely unreliable
> To a degree that I had to duplicate them, my app sends its own callbacks.
> What is unreliable you may ask? Disconnect notification for example may come
> *minutes* after. So my client already handled the situation and connected to
> another channel long ago. If I would have waited for presence notification,
> I would have lost the user. Or even worse on some days the notification may
> not come at all.
> And because you can never be totally sure that you have not missed something
> or channel doesn't give you a new trick, I had to add shamefull lazy polling
> to make sure my users are still connected :(

Is it similar to
http://code.google.com/p/googleappengine/issues/detail?id=5975  or
something else?

Hmm not sure. Consider the following use case: client connects to a channel, some time later he goes offline and comes back, client sees that the token is still valid and reconnects, so far so good. Now on the server I'm tracking the list of connected users. Obviously what I expect is a connected presence, then when user goes offline - disconnected and finally when he is reconnected - connected again. In practice it could easily be connected, again connected and then disconnected. When that disconnected presence comes I have no way of knowing if this one  is the missing between 2 previous connected presences or this one is fresh and the missing one is still coming. Maybe add some kind of counter that would be increased with each presence?

cheers,

Aleksei

[Aleksei Rovenski]

> PS. One final thing I forgot to mention about Channel API problems is that
> it seems some of the users simply can not load the channel javascript lib
> which is hosted by google... What the hell...

Are you refererring to:
http://code.google.com/p/googleappengine/issues/detail?id=4764

Or something else?

Hmm I don't think so.  Before client tries to create a channel it first checks if the script has been loaded, basically same way as this module does:

http://code.google.com/p/gwt-gae-channel/

And it turns out that 4% do not have it. I have a gwt app, so at this point it is obvious that javascript is supported. In such cases I simply readd the script to the header and log what happens. Most of the time nothing happens, but a tiny portion of users load end up having the script loaded, so it doesn't look like 4764 as that one I guess is permanent or maybe it is a mix of issues here. 

I remember there was an issue long time ago when channel script request returned empty response, and afaik that bug has been closed in the tracker, but maybe it hasn't been fully fixed.

cheers,

Aleksei

[Kristopher Giesing]

I thought the Channel API was already based on web sockets (for platforms that support them).

- Kris

[Kristopher Giesing]

> 3. The implementation is different from the test and production server

Care to elaborate? (file new Defect if necessary)

It's pretty clear to anyone working with the system more than casually that the implementations are different.  I'm not sure that's really a problem; I mean, the whole dev server is different from the production server.  There's only so much you can mimic from a distributed hosting environment locally.

What *is* a problem is when the behaviors differ subtly from dev to prod.  For example, if the dev server guarantees (even if accidentally) event ordering in one way, when in prod the event ordering isn't guaranteed, that's a problem.  I'm not familiar enough with channels in prod to know what might fall into that category, but I've seen very similar effects in other systems.

- Kris 

[Pascal Patry]

On September 3, 2012 10:52:55 Johan Euphrosine wrote:
> On Fri, Aug 31, 2012 at 9:28 PM, Pascal Patry <ppa...@spectrumdt.com> wrote:
> > Hi,
> >
> > Channels on AppEngine are picky. There are multiple issues that I had to
> > work around for an application that is currently in production:
>
> Thanks for your feedback, please make sure you fill new
> defects/feature requests or star existing ones about those issues so
> you can be notified about our progress:
> http://code.google.com/p/googleappengine/issues/list?cursor=googleappengine%3A7718&q=Component%3DChannel
>
> > 1. Channels aren't working on backends
>
> http://code.google.com/p/googleappengine/issues/detail?id=5123
>
> > 2. No API is available to mobile
>
> http://code.google.com/p/googleappengine/issues/detail?id=4226
> http://code.google.com/p/googleappengine/issues/detail?id=4189
>
> > 3. The implementation is different from the test and production server
>
> Care to elaborate? (file new Defect if necessary)

The way this is working on GAE is different from the local Jetty
instance. It's pretty well known since it's piggybacking XMPP/GChat
infrastructure so I do understand that it would be hard to do the same
on local instances... but the fact that those are different doesn't help
at all.

> > 4. Connection/Disconnection handlers aren't reliably called
>
> Does this correspond to this?
> http://code.google.com/p/googleappengine/issues/detail?id=5975

Thanks for letting us know about that issue as well :)

What I was talking is on the server-side when the
Connection/Disconnection presence handlers are called. We had an
application initializing the context for each user once they were
connecting and found out that they weren't called that reliably. So we
moved the code to use lazy initialization.

[Aleksei Rovenski]

Hmm not sure. Consider the following use case: client connects to a channel, some time later he goes offline and comes back, client sees that the token is still valid and reconnects, so far so good. Now on the server I'm tracking the list of connected users. Obviously what I expect is a connected presence, then when user goes offline - disconnected and finally when he is reconnected - connected again. In practice it could easily be connected, again connected and then disconnected. When that disconnected presence comes I have no way of knowing if this one  is the missing between 2 previous connected presences or this one is fresh and the missing one is still coming. Maybe add some kind of counter that would be increased with each presence?

I have added a feature request to add timestamp for channel presences:

http://code.google.com/p/googleappengine/issues/detail?id=8151

Please star this issue! 

cheers,

Aleksei