Retry-After

[John Pignata]

Hey All,

I'm working on implementing a Google Cloud Messaging server-side sender and had a question about retries. In the architectural overview it's noted that any Retry-After header passed back from the server must be respected. In the HTTP spec it's stated that the field can either return an HTTP date or a delta to wait in seconds. Is it documented anywhere which I can expect from the GCM endpoint? Is there anyway to trigger this condition for testing purposes?

Thanks,

-jp

[Skip Morrow]

Felipe,

I would also like the answer to this.

By the way, have you considered building a GCM sandbox where we could ask the server to send us errors and whatnot so we can test our apps against them? Thanks.

[Trevor Johns]

John: The HTTP specification allows both, so your server code needs to be designed to handle both. We design to comply with standards, we need app developers to do the same. This gives us the flexibility needed to update our code safely in the future to provide new services.

As the saying goes, "be strict when sending and tolerant when receiving." (RFC 1958, Architectural Principles of the Internet)

Skip: The 'dry_run' parameter will do what you want. Documentation is here: http://developer.android.com/guide/google/gcm/gcm.html#request

[Skip Morrow]

Trevor, thanks, but what I was after was a way to have google send me responses with different errors so I can test my app server.

For instance, how would I get the following http response codes in a response?

200: No problem

400: I can send a message with an invalid JSON. No problem

401: I could send a made up developer key. No problem

500: cannot simulate

503: cannot simulate. This would also include the retry-after header, as the OP requested

Furthermore, the following errors cannot be simulated for testing

Unavailable

NotRegistered

Canonical id's

MismatchedSender???

Maybe others...

I was envisioning a sandbox of sorts where we pre-configure it to send certain responses for testing. We then send the message to that sandbox server and the server responds accordingly. I'm pretty sure the creative folks at google could think of the best way to implement something like this.

[Trevor Johns]

Skip,

This is what unit testing and mocks are for. It's not hard to set an arbitrary response code in a mock response. On the other hand, it takes quite a bit of work to implement on the server.

If you don't have enough documentation to implement this, that's something that should be addressed. In which case, let us know.

--

You received this message because you are subscribed to the Google Groups "android-gcm" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-gcm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/android-gcm/-/Z7nq6JZUT0sJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Trevor Johns
Google Developer Programs, Android
http://developer.android.com

[Blaine Sanders]

Trevor,

I feel the documentation is lacking regarding the structure of the JSON reply from GCM. Certainly, if we must honor the Retry-After header when included in the reply, we either need to see the structure documented so we know how to parse it, or we need to be able to cause the condition. We can simulate the condition only if we know how to build the response data.

Based on the documentation, all I can postulate is that the response will look something like this:

"{"Retry-After":???,"multicast_id":5257598644491863635,"success":0,"failure":1,"canonical_ids":0,"results":[{"error":"Unavailable"}]}"

But 1) that doesn't seem right, because all other keys are lowercase-underscore, and 2) I don't know what the value of Retry-After is supposed to be, because it's not documented. I'm guessing its a number of seconds to wait before starting the exponential backoff...? But guessing isn't enough. We need more documentation.

Thanks,

Blaine

[Felipe Leme]

Blaine,

The Retry-After is not part of the JSON (which is sent as the HTTP body), but it's an HTTP header. Regarding the format, we are not specific because the W3 specification allows both a date or an interval:

14.37 Retry-After

The Retry-After response-header field can be used with a 503 (Service Unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. This field MAY also be used with any 3xx (Redirection) response to indicate the minimum time the user-agent is asked wait before issuing the redirected request. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response.

       Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of its use are

       Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
       Retry-After: 120

In the latter example, the delay is 2 minutes.

If we stick to one of them (and document it) but have to change later, it would break existing apps, so it's safer to assume that both are possible.

Hope if helps,

-- Felipe

[Blaine Sanders]

Ah, thank you Felipe. That clarifies things.

So, in my case, I am opening my connection and sending my request to GCM server in php via cURL. I send my request parameters as a JSON-encoded string, and my response is returned in JSON on success.

Is there an equivalent to Retry-After in the JSON response?

Currently, whenever I get an error response of 'Unavailable' or 'InternalServerError', I initiate an exponential backoff and retry 5 times.

Is there anything else I need to do to avoid potential blacklisting?

Thanks,

Blaine

[Blaine Sanders]

I think I found the answer to my question. I need to return both the headers and the body. The problem is, in order to do so I need to set these curl options:

CURLOPT_RETURNTRANSFER = true

CURLOPT_HEADER = true

The problem is, that gives me everything back combined into one string that I then have to parse with a regex to separate the headers and the json. Then I have to parse the header to get the status code and possible Retry-After value. What a pain! Is there no other way?

[Trevor Johns]

Rather than using curl, you could use the HTTP library that ships with Java/Ruby/Python/your language of choice. These libraries usually make it much easier to parse headers.

--

You received this message because you are subscribed to the Google Groups "android-gcm" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-gcm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/android-gcm/-/xhrZ3XTTfrIJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Blaine Sanders]

Thanks Trevor. I'm working on a php framework, and there's no built-in library for dealing with http headers. I can install some PECL library for dealing with it, but I've opted to use a modified parsing function that I've gleaned off the internet, rather than force the entire development team to install a large library locally as well as on the server just to deal with one simple operation.

I need a little more clarity on exactly how timeouts should be handled. If I'm understanding correctly, I need to examine my http response body and determine if there are any errors of the type 'Unavailable' or 'InternalServerError'. If so, I need to examine the http response header and determine if there is a Retry-After. If there is, I need to determine either from the timestamp or the number of seconds how long I need to wait before making another attempt. On that successive attempt, I also need to back off exponentially from the previous attempt.

Questions:

1. Do I need to honor the Retry-After header on each retry, or is it just on the first retry, then exponentially back off?

2. Do I need to combine the Retry-After with the exponential backoff, or does the Retry-After value trump the backoff value?

3. Will Retry-After always be sent with an 'Unavailable'/'InternalServerError', or is it just sometimes?

[Felipe Leme]

Hi Blaine,

See answers inline...

let's say that your first post returned Unavailable without a Retry-After

On Wed, Nov 28, 2012 at 9:32 AM, Blaine Sanders <blaines...@gmail.com> wrote:

1. Do I need to honor the Retry-After header on each retry, or is it just on the first retry, then exponentially back off?

You need to honer the Retry-After only when you received it; if when you retry you didn't get a header, you could continue your back-off sequence.

 

2. Do I need to combine the Retry-After with the exponential backoff, or does the Retry-After value trump the backoff value?

The Retry-After should trump. Let's say you would retry in 60 seconds but the Retry-After says 100 seconds, you should wait at least 100 seconds.

 

3. Will Retry-After always be sent with an 'Unavailable'/'InternalServerError', or is it just sometimes?

No, it's not always sent.

Here's an example, in a scenario where your algorithm would back off for 30, 60, 120, then 240 seconds (these are just random numbers I picked up):

1. Post returns Unavailable without Retry-After.
2. 30 seconds later you post again, it returns Unavailable but with a Retry-After of 100 seconds. Now instead of waiting 60 seconds, you should wait 100.
3. 100 seconds later, you post again, and it still returns Unavailable with a Retry-After of 100 seconds. At this point, you could wait 100 or 120 seconds (which is your next back-off value).

Generally speaking, if you are doing the exponential back-off right, you might never get a Retry-After header. So, if it's too complicate to implement the Retry-After logic in your server, I wouldn't worry about it - as long as your server is behaving well and doing exponential back-off, it won't be blacklisted.

 -- Felipe

[Blaine Sanders]

Thank you very much Felipe! That was very helpful.

The only remaining question I have now is, in your example you gave 30, 60, 120, and 240 as your example (which you duly noted as being purely theoretical), but the documentation implies that 1, 2, 4, 8 would be perfectly acceptable:

"e.g. if you waited one second before the first retry, wait at least two second before the next one, then 4 seconds and so on"

Of course, 1, 2, 4, 8 would be much more appealing than 30, 60, etc... and if that's acceptable to the GCM server, that's what I'd prefer to implement. I will likely implement some logic to queue up severely delayed notifications and retry them with some form of cron mechanism, but would definitely prefer not to have to handle those situations if I can avoid them occurring.

Here is an example of how I might handle retries:

1. Post returns Unavailable without Retry-After.
   

1. 2 seconds later, post again, returns Unavailable without Retry-After
2. 4 seconds later, post again, returns Unavailable but with a Retry-After of 180 seconds.
3. Since retry is several minutes in the future, kill retry sequence, derive a retry timestamp, and queue notification into db
4. check db with cron job every five minutes and try to re-send notifications whose retry stamp have passed.

I really don't want my server spinning wait cycles for minutes at a time when there are possibly hundreds or thousands of delayed notifications piling up. I'd rather free up the server resources and try again later. Do you see any potential problems with this solution?

[Trevor Johns]

Blane,

I recommend looking at Zend Framework's HTTP client (Zend_Http_Client).

I did some work on it back in the day. It will use lib_curl if it's installed (I recommend it), but I believe it has a fallback sockets-based implementation that should run on pretty much any PHP environment without extra extensions.

http://framework.zend.com/

I'll let Felipe answer your questions about timeout values. :)

On Wednesday, November 28, 2012, Blaine Sanders wrote:

Thanks Trevor. I'm working on a php framework, and there's no built-in library for dealing with http headers. I can install some PECL library for dealing with it, but I've opted to use a modified parsing function that I've gleaned off the internet, rather than force the entire development team to install a large library locally as well as on the server just to deal with one simple operation.

I need a little more clarity on exactly how timeouts should be handled. If I'm understanding correctly, I need to examine my http response body and determine if there are any errors of the type 'Unavailable' or 'InternalServerError'. If so, I need to examine the http response header and determine if there is a Retry-After. If there is, I need to determine either from the timestamp or the number of seconds how long I need to wait before making another attempt. On that successive attempt, I also need to back off exponentially from the previous attempt.

Questions:

1. Do I need to honor the Retry-After header on each retry, or is it just on the first retry, then exponentially back off?

2. Do I need to combine the Retry-After with the exponential backoff, or does the Retry-After value trump the backoff value?

3. Will Retry-After always be sent with an 'Unavailable'/'InternalServerError', or is it just sometimes?

--

You received this message because you are subscribed to the Google Groups "android-gcm" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-gcm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/android-gcm/-/J8KwaFa0BZEJ.

For more options, visit https://groups.google.com/groups/opt_out.