Sprint 4G(WIMAX) tcp connection problem

[Long]

Hi Developers, 

    I have an background service which connects my server with a TCP connection.

    My background service sends a ping packet to my server every 4minutes to keep the socket alive.

   The problem is that the device can not get the packet from the server after a short time(3 seconds from my test) after the ping is sent. 

   This app works fine on other networks(AT&T, verizon). 

 

   The strange thing is that if I force the device to connect to 3G(EVDO revision A), it works fine.

 

   It seems to me that, on Sprint 4G network, when an app sends a packet through a tcp socket, and then the socket is idle for

several seconds, then the app will never be able to receive packets from this socket.

   

   I don't think it's related to the 4G signal, since this issue can be reproduced every time.

   Any suggestions?

 

   

  

   

    

[Robert Greenwalt]

Could you use GCM instead of rolling your own long-lived connection with independent keepalive?  It'll be better for the users battery and better for the carriers network and you won't have to figure out issues like these.

 

   

  

   

    

--
--
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-d...@googlegroups.com
To unsubscribe from this group, send email to
android-develop...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
---
You received this message because you are subscribed to the Google Groups "Android Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-develop...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kostya Vasilyev]

Robert,

Could you elaborate on the "issues like these" part? Are there changes in Android that developers should be made aware?

I also have an app that uses long persistent server connection, and using GCM is not an option either.

It seems that with Android 4.1 and 4.2, connections often go "dead" -- even though technically connected (at the server side), data pushed by the server stops getting through (this is verified by server logs).

Android 4.0 does not seem to suffer from these issues, at least not to the same extent.

A Nexus 7 and a Galaxy Nexus running the same stock 4.2.2, connected to the same server, over the same WiFi network, will experience dead connections at different rates (confirmed by logging).

The suggestion to use GCM does not benefit everyone.

It's nice that that Android has GTalk which provides GCM, and has its own keep-alive logic, but with it being closed source, and with apparent changes in recent Android versions, this leaves those developers who can't use GCM dead in the water.

It would be great if someone from Google could comment, and explain, what's different in Android 4.1 - 4.2 with respect to long running connections, and what sort of magic is now required to keep them alive, for those apps that really need it.

Thanks,

-- K

2013/4/15 Robert Greenwalt <rgree...@google.com>

[bob]

What port number are you using on the server?

Maybe use port 80 as it is more open?

Thanks.

[Robert Greenwalt]

On Mon, Apr 15, 2013 at 8:44 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

Robert,

Could you elaborate on the "issues like these" part? Are there changes in Android that developers should be made aware?

By "issues like these" I just meant the inevitable bugs that will crop up when making software.  If there is a library on the platform that already does what you need it is risky to decide to create your own redundant code.

I also have an app that uses long persistent server connection, and using GCM is not an option either.

Can you elaborate on why GCM is not an option for you?  We want to make GCM useful, so if it's missing features or "is not an option" we want to know about it..

 

It seems that with Android 4.1 and 4.2, connections often go "dead" -- even though technically connected (at the server side), data pushed by the server stops getting through (this is verified by server logs).

Is that just on WiMAX as the original poster suggested?  Do you have bugreports and/or packet captures?

 

Android 4.0 does not seem to suffer from these issues, at least not to the same extent.

A Nexus 7 and a Galaxy Nexus running the same stock 4.2.2, connected to the same server, over the same WiFi network, will experience dead connections at different rates (confirmed by logging).

Can you share a bugreport and packet capture?  There was no intention to kill long-lived connections in android, so if there is an issue with 4.2.* we need to hear about it.

The suggestion to use GCM does not benefit everyone. 

It's nice that that Android has GTalk which provides GCM, and has its own keep-alive logic, but with it being closed source, and with apparent changes in recent Android versions, this leaves those developers who can't use GCM dead in the water.

It would be great if someone from Google could comment, and explain, what's different in Android 4.1 - 4.2 with respect to long running connections, and what sort of magic is now required to keep them alive, for those apps that really need it.

As I said above we did not intentionally do anything to break long-lived connections.  We encourage GCM use as it lets us do optimizations based on mobile network type, etc to save on signaling traffic and battery life, but you should still be able to roll your own if you wish.

There was a bug introduced in the 4.1 timeframe on mobile-data connections we stopped counting udp packets when determining if a mobile connection was hung - this could cause us to reset the mobile connection even when it was live if only udp packets were being passed.  This doesn't apply to wifi OR to mobile with GTalk alive - only on a private mobile network where GTalk servers are inaccessible.  

[Kostya Vasilyev]

2013/4/15 Robert Greenwalt <rgree...@google.com>

On Mon, Apr 15, 2013 at 8:44 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

Robert,

Could you elaborate on the "issues like these" part? Are there changes in Android that developers should be made aware?

By "issues like these" I just meant the inevitable bugs that will crop up when making software.  If there is a library on the platform that already does what you need it is risky to decide to create your own redundant code.

Yes, I understand the inevitable bugs, but GTalk / GCM work with those bugs, perhaps becuase they do things a certain way.

I've seen GTalk's service menu, it seems it has separate keep-alive logic for mobile vs. WiFi.... Why? I can't tell just by reading the Android networking docs.

 

I also have an app that uses long persistent server connection, and using GCM is not an option either.

Can you elaborate on why GCM is not an option for you?  We want to make GCM useful, so if it's missing features or "is not an option" we want to know about it..

I have a little email app.

Unless someone can convince Gmail, Yahoo, GMX, AOL, Dovecot, Courier, and all the rest switch to GCM, I'm kind of stuck with IMAP IDLE, which is a terrible protocol, but at least it's there.

 

 

It seems that with Android 4.1 and 4.2, connections often go "dead" -- even though technically connected (at the server side), data pushed by the server stops getting through (this is verified by server logs).

Is that just on WiMAX as the original poster suggested?  

I don't have / use WiMax, the case I and one of my users investigated with logging was WiFi.

 

Do you have bugreports and/or packet captures?

I have seen server logs captured by a very helpful user, and ran my own app with logging, on two devices with 4.2.2, connected to the same WiFi access point, and same mail server.

The symptoms are -- I can outline them in just a few sentences.

My Android code is in a socket read (in a worker thread, of course), the device goes to sleep, the connection stays up.

The server sends small packets of data from time to time.

When the device has been asleep for a while (5-8-10 minutes, IIRC) and the server sends data:

- The app no longer wakes up and receives the data

- The next socket write from the server results in a socket write error (so the first packet was not acknowledged)

In the tests I ran, the connection was alive at my home router (it has a UI where I can see the connections).

If the device was not allowed to go to sleep, the issue did not occur.

 

 

Android 4.0 does not seem to suffer from these issues, at least not to the same extent.

A Nexus 7 and a Galaxy Nexus running the same stock 4.2.2, connected to the same server, over the same WiFi network, will experience dead connections at different rates (confirmed by logging).

Can you share a bugreport and packet capture?  There was no intention to kill long-lived connections in android, so if there is an issue with 4.2.* we need to hear about it.

Don't have a packet capture, but will try to submit a bug report, since there seems to be interest on your part. Thank you for this.

Are there any changes to WiFi power saving in 4.1 - 4.2, perhaps it tries to go deeper than before into some sort of sleep state?

Is it possible that this power saving mode has a low-level bug, where the WiFi radio stops responding to incoming packets?

Would this be related to this old issue? I can only guess of course: https://code.google.com/p/android/issues/detail?id=9781

 

The suggestion to use GCM does not benefit everyone. 

It's nice that that Android has GTalk which provides GCM, and has its own keep-alive logic, but with it being closed source, and with apparent changes in recent Android versions, this leaves those developers who can't use GCM dead in the water.

It would be great if someone from Google could comment, and explain, what's different in Android 4.1 - 4.2 with respect to long running connections, and what sort of magic is now required to keep them alive, for those apps that really need it.

As I said above we did not intentionally do anything to break long-lived connections.  We encourage GCM use as it lets us do optimizations based on mobile network type, etc to save on signaling traffic and battery life, but you should still be able to roll your own if you wish.

How about documenting those "optimizations, etc.", or specifically, the special things GTalk has to do becuase it runs on Android?

GTalk/GCM seem to work well, but is closed source. I assume there some subtle parts, which benfit from knowing what the system does (something more than "create a socket, and call connect() to connect...").

 

 

There was a bug introduced in the 4.1 timeframe on mobile-data connections we stopped counting udp packets when determining if a mobile connection was hung - this could cause us to reset the mobile connection even when it was live if only udp packets were being passed.  This doesn't apply to wifi OR to mobile with GTalk alive - only on a private mobile network where GTalk servers are inaccessible.  

Yes, I remember that discussion, and it doesn't apply in my case.

-- K

[Irfan Sheriff]

On Mon, Apr 15, 2013 at 10:46 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

2013/4/15 Robert Greenwalt <rgree...@google.com>

On Mon, Apr 15, 2013 at 8:44 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

Robert,

Could you elaborate on the "issues like these" part? Are there changes in Android that developers should be made aware?

By "issues like these" I just meant the inevitable bugs that will crop up when making software.  If there is a library on the platform that already does what you need it is risky to decide to create your own redundant code.

Yes, I understand the inevitable bugs, but GTalk / GCM work with those bugs, perhaps becuase they do things a certain way.

I've seen GTalk's service menu, it seems it has separate keep-alive logic for mobile vs. WiFi.... Why? I can't tell just by reading the Android networking docs.

I dont know what gtalk is doing, but a guess here is that wifi needs more aggressive keep alives due to NAT time outs being low on some access points.

 

 

I also have an app that uses long persistent server connection, and using GCM is not an option either.

Can you elaborate on why GCM is not an option for you?  We want to make GCM useful, so if it's missing features or "is not an option" we want to know about it..

I have a little email app.

Unless someone can convince Gmail, Yahoo, GMX, AOL, Dovecot, Courier, and all the rest switch to GCM, I'm kind of stuck with IMAP IDLE, which is a terrible protocol, but at least it's there.

 

 

It seems that with Android 4.1 and 4.2, connections often go "dead" -- even though technically connected (at the server side), data pushed by the server stops getting through (this is verified by server logs).

Is that just on WiMAX as the original poster suggested?  

I don't have / use WiMax, the case I and one of my users investigated with logging was WiFi.

 

Do you have bugreports and/or packet captures?

I have seen server logs captured by a very helpful user, and ran my own app with logging, on two devices with 4.2.2, connected to the same WiFi access point, and same mail server.

The symptoms are -- I can outline them in just a few sentences.

My Android code is in a socket read (in a worker thread, of course), the device goes to sleep, the connection stays up.

The server sends small packets of data from time to time.

When the device has been asleep for a while (5-8-10 minutes, IIRC) and the server sends data:

- The app no longer wakes up and receives the data

- The next socket write from the server results in a socket write error (so the first packet was not acknowledged)

In the tests I ran, the connection was alive at my home router (it has a UI where I can see the connections).

If the device was not allowed to go to sleep, the issue did not occur.

 

 

Android 4.0 does not seem to suffer from these issues, at least not to the same extent.

A Nexus 7 and a Galaxy Nexus running the same stock 4.2.2, connected to the same server, over the same WiFi network, will experience dead connections at different rates (confirmed by logging).

Can you share a bugreport and packet capture?  There was no intention to kill long-lived connections in android, so if there is an issue with 4.2.* we need to hear about it.

Don't have a packet capture, but will try to submit a bug report, since there seems to be interest on your part. Thank you for this.

Are there any changes to WiFi power saving in 4.1 - 4.2, perhaps it tries to go deeper than before into some sort of sleep state?

Is it possible that this power saving mode has a low-level bug, where the WiFi radio stops responding to incoming packets?

Would this be related to this old issue? I can only guess of course: https://code.google.com/p/android/issues/detail?id=9781

This bug is a bit old. 

Can you clarify what  android device and access point you are using. You could send Robert or me an email with the details and the bugreport (a packet capture showing the problem may be even better)

I ask because different wifi drivers do their own power optimizations and I have seen issues where these optimizations do not work well with certain access points. We have to chase these issues with the vendors.

There should not be generic issue like this on the platform itself - it is depended on the wifi driver.

Thanks 

 

The suggestion to use GCM does not benefit everyone. 

It's nice that that Android has GTalk which provides GCM, and has its own keep-alive logic, but with it being closed source, and with apparent changes in recent Android versions, this leaves those developers who can't use GCM dead in the water.

It would be great if someone from Google could comment, and explain, what's different in Android 4.1 - 4.2 with respect to long running connections, and what sort of magic is now required to keep them alive, for those apps that really need it.

As I said above we did not intentionally do anything to break long-lived connections.  We encourage GCM use as it lets us do optimizations based on mobile network type, etc to save on signaling traffic and battery life, but you should still be able to roll your own if you wish.

How about documenting those "optimizations, etc.", or specifically, the special things GTalk has to do becuase it runs on Android?

GTalk/GCM seem to work well, but is closed source. I assume there some subtle parts, which benfit from knowing what the system does (something more than "create a socket, and call connect() to connect...").

 

 

There was a bug introduced in the 4.1 timeframe on mobile-data connections we stopped counting udp packets when determining if a mobile connection was hung - this could cause us to reset the mobile connection even when it was live if only udp packets were being passed.  This doesn't apply to wifi OR to mobile with GTalk alive - only on a private mobile network where GTalk servers are inaccessible.  

Yes, I remember that discussion, and it doesn't apply in my case.

-- K

[Robert Greenwalt]

u

On Mon, Apr 15, 2013 at 10:46 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

2013/4/15 Robert Greenwalt <rgree...@google.com>

On Mon, Apr 15, 2013 at 8:44 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

Robert,

Could you elaborate on the "issues like these" part? Are there changes in Android that developers should be made aware?

By "issues like these" I just meant the inevitable bugs that will crop up when making software.  If there is a library on the platform that already does what you need it is risky to decide to create your own redundant code.

Yes, I understand the inevitable bugs, but GTalk / GCM work with those bugs, perhaps becuase they do things a certain way.

I've seen GTalk's service menu, it seems it has separate keep-alive logic for mobile vs. WiFi.... Why? I can't tell just by reading the Android networking docs.

Most ipv4 is behind nats.  Even on the carrier side.  Each NATing agent will have a timeout - no activity and they forget the connection state and your connection is broken.  That timeout varies form network to network and the timeout GTalk or GCM use have been tuned from android experience.  This sort of tuning is another reason why relying on GCM is better than trying to reinvent it.

[Kostya Vasilyev]

2013/4/15 Irfan Sheriff <ishe...@gmail.com>

On Mon, Apr 15, 2013 at 10:46 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

I've seen GTalk's service menu, it seems it has separate keep-alive logic for mobile vs. WiFi.... Why? I can't tell just by reading the Android networking docs.

I dont know what gtalk is doing, but a guess here is that wifi needs more aggressive keep alives due to NAT time outs being low on some access points.

I could see the connections still alive in my router's UI.

Not letting the device go to sleep would make the issue go away (with no change at the router or NAT or server).

Let me ask you this -- when WiFi goes into power saving mode, is there a specific upper bound on the time it takes to wake back up (either triggered by an incoming packet, or an outgoing one)?

 

 

Don't have a packet capture, but will try to submit a bug report, since there seems to be interest on your part. Thank you for this.

Are there any changes to WiFi power saving in 4.1 - 4.2, perhaps it tries to go deeper than before into some sort of sleep state?

Is it possible that this power saving mode has a low-level bug, where the WiFi radio stops responding to incoming packets?

Would this be related to this old issue? I can only guess of course: https://code.google.com/p/android/issues/detail?id=9781

This bug is a bit old. 

It's the closest I could find, sorry. I vaguely recall seeing some newer ones about the same thing.

 

Can you clarify what  android device and access point you are using. You could send Robert or me an email with the details and the bugreport (a packet capture showing the problem may be even better)

I do not have a packet capture, I had server and app logs, and given that there are no packet captures, I already explained the gist of it (saving you having to read the logs).

My devices were (and are) a Galaxy Nexus and a Nexus 7 with 4.2.2 (or 4.2.1?, I worked on this a lot in mid-December of last year). The user's devices included a Galaxy Nexus with 4.1.

My home router, at the time, was a Zyxel Keentic Giga, with an 80 MBit/s PPTP connection.

 

I ask because different wifi drivers do their own power optimizations and I have seen issues where these optimizations do not work well with certain access points. We have to chase these issues with the vendors.

Ok, so the bug is a bit old, but the issue remains to this day?

I can appreciate the complexity of tracking these down, and glad someone is doing it, but...

....it does not help when users have but the latest Android version with the latest drivers with fixes (I presume).

 

There should not be generic issue like this on the platform itself - it is depended on the wifi driver.

Looking at it from an application's point of view, it makes no difference if it's the driver or the framework or the kernel code.

To the user it's even more simple: "does not work".

Should there be a system level setting to turn off those power optimizations, if they're known to not be compatible with "all" (taken as a metaphorical term) access point / router devices?

-- K

[Irfan Sheriff]

There is. You could try turning off settings > wifi > advanced > wi-fi optimization

 to see if it makes a difference.

The HIGH_PERF API also does the same and should be used with care due to power impact.

[Robert Greenwalt]

I think he's suggesting that if app developers can't rely on wifi not losing packets on this hardware with the AP it will see in the wild, we should disable optimizations in general.

I think we found from our testing that the optimizations worked with a high percentage of AP - all AP we tested against before shipping.

[Kostya Vasilyev]

2013/4/15 Robert Greenwalt <rgree...@google.com>

Most ipv4 is behind nats.  Even on the carrier side.  Each NATing agent will have a timeout - no activity and they forget the connection state and your connection is broken.

Keeping the device from going into deep sleep helped avoid the issue. The NATs can't possibly know that the Android device at the end of a connection has its screen on or off.

So, I have to conclude it had to do with WiFi power saving.

Is there a specific time limit for the WiFi hardware to come out of deep power save state?

Knowing this would let me use proper timeouts in my keepalive code. Could you share this information?

 

 That timeout varies form network to network and the timeout GTalk or GCM use have been tuned from android experience.  This sort of tuning is another reason why relying on GCM is better than trying to reinvent it.

IMAP / GCM? No, really, I see your point, but there are times when GCM is not an option.

 

-- K

[Kostya Vasilyev]

2013/4/15 Irfan Sheriff <ishe...@gmail.com>

On Mon, Apr 15, 2013 at 11:50 AM, Kostya Vasilyev <kman...@gmail.com> wrote:

Should there be a system level setting to turn off those power optimizations, if they're known to not be compatible with "all" (taken as a metaphorical term) access point / router devices?

There is. You could try turning off settings > wifi > advanced > wi-fi optimization

 to see if it makes a difference.

Thank you. I never noticed that before. Must be new in 4.2?

 

The HIGH_PERF API also does the same and should be used with care due to power impact.

Can you give an estimate how much more power this will consume?

-- K

[Irfan Sheriff]

I do not have the numbers with me now, but this really depends on the driver and the environment (amount of traffic on air)

I believe in our tests at google, the idle power went up 5 times. 

-- K

[Long Pu]

I am connecting to tcp port 5222 which provides xmpp service

--

--
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-d...@googlegroups.com
To unsubscribe from this group, send email to
android-develop...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Android Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-developers/3FvPZp5n2K0/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to android-develop...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Things don' happen. Things are made to happen.

[Long Pu]

We have to consider multiple platforms(iOS, Android, Blackberry). On iOS, push notification

is not an option for our application.  If use GCM, we need  another extra special design especially for android.

You received this message because you are subscribed to a topic in the Google Groups "Android Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-developers/3FvPZp5n2K0/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to android-develop...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Greenwalt]

Thanks Long Pu for the reasoning for not using GCM.

You only have this problem on sprint wimax, correct?  Have to tried very frequent keep-alives?  Figuring out roughly what the required period is may help us find the cause.  We can talk with Sprint and Samsung about it, but the timeout info would be useful before that.

R

[bob]

Is it an option to run the XMPP server on port 80?

Or perhaps just set up a test server on port 80 to explore the hypothesis that this will resolve the problem?

Alternatively, you could also create a proxy that runs on port 80 and acts as an intermediary.

Thanks.

On Wednesday, April 17, 2013 9:20:27 AM UTC-5, Long wrote:

I am connecting to tcp port 5222 which provides xmpp service

On Tue, Apr 16, 2013 at 12:00 AM, bob <b...@coolfone.comze.com> wrote:

What port number are you using on the server?

Maybe use port 80 as it is more open?

Thanks.

On Saturday, April 13, 2013 5:35:13 AM UTC-5, Long wrote:

Hi Developers, 

    I have an background service which connects my server with a TCP connection.

    My background service sends a ping packet to my server every 4minutes to keep the socket alive.

   The problem is that the device can not get the packet from the server after a short time(3 seconds from my test) after the ping is sent. 

   This app works fine on other networks(AT&T, verizon). 

 

   The strange thing is that if I force the device to connect to 3G(EVDO revision A), it works fine.

 

   It seems to me that, on Sprint 4G network, when an app sends a packet through a tcp socket, and then the socket is idle for

several seconds, then the app will never be able to receive packets from this socket.

   

   I don't think it's related to the 4G signal, since this issue can be reproduced every time.

   Any suggestions?

 

   

  

   

    

--
--
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-d...@googlegroups.com
To unsubscribe from this group, send email to

android-developers+unsub...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Android Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-developers/3FvPZp5n2K0/unsubscribe?hl=en.

To unsubscribe from this group and all its topics, send an email to android-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Greenwalt]

I agree - testing 80 sounds like a good idea to gain more info about what may be happening.

android-develop...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
---

You received this message because you are subscribed to the Google Groups "Android Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-develop...@googlegroups.com.

[Long Pu]

We only see this issue on Boost phones on WIMAX actually. Pure Sprint phones work fine.   

The minimal required period is a crazy value like 3~5 seconds.   Hope this info can help.

Our guess is that maybe Boost rents other company's WIMAX network, so they would like to try their best to clean up idle connections. 

Thanks.

Cheers.

Long

[Robert Greenwalt]

What phone hardware?

[Long Pu]

It's galaxy S2. 

I believe the GCM works on this phone. I am wondering what GCM does to keep his socket alive.

It would not seed keep alive packets in short period like several seconds, this would burn

out battery very quickly.

My guess is that the carrier can recognize the GCM tcp connection and only keep it alive.

Cheers.

Long

[Robert Greenwalt]

It is possible the samsung radio is GCM aware.  That would be bad.  I've added an internal bug to add a CTS test for this.