Events-List started returning 410 when using updatedMin of yesterday

[Eyal Yavor]

Hey,

I've been using Events-List with the updatedMin to track changes in some calendars.

Code was working great, until two days ago, where in some calendars I started receiving 410 response when using updatedMin.

I know that usually the limit is 20 days, but now it seems it's down to 11-12 hours.

Anyone ran into this as well?

Thanks,

Eyal

{"error": {"errors": [

  {

   "domain": "calendar",

   "reason": "updatedMinTooLongAgo",

   "message": "The requested minimum modification time lies too far in the past.",

   "locationType": "parameter",

   "location": "updatedMin"

  }

 ],

 "code": 410,

 "message": "The requested minimum modification time lies too far in the past."

}

}

[Lucia Fedorova]

Hi Eyal,

it is recommended to switch away from updatedMin and use sync tokens instead: 

https://developers.google.com/google-apps/calendar/v3/sync

[Yannick Biet]

Hi Lucia,

In this scenario, Google acts like the master.

But as we are integrating meetings from a CRM database, we are unable to wipe out all CRM data (each crm meeting has additionnal business/critical information not synchronized to Google)

So the scenario of the synctoken is really complex as when we will get  410 errors, we should have to sync back all the meetings in the Google Calendar... that could be "a lot" of meetings for nothing

(some of our customer base have dozens of thousands of meetings !)

In V2, this parameter has never been a problem !

[Lucia Fedorova]

Hello Yannick,

In V2 it was possible that the data between the server and the client would get desynchronized and it would not be detected. It was for example possible to get into the state where events would be deleted from the server but would still be present in the client. V3 has stronger consistency guarantees than V2 did, which is why you need to handle 410s.

However, wiping the storage is only one of possible ways to handle it. You can for example choose to only re-sync the time-range which you are interested in (and for which you want to be consistent) and from then on you can continue performing incremental synchronization.

[Frank C]

We're seeing this too.  Something sure seems to have changed in the last few days.  We're getting updatedMinTooLongAgo now a lot more often as well... and sometimes it happens moments after it succeeded.

For example we made the following request at 2014-11-19T16:10 -08:00.   

GET https://www.googleapis.com/calendar/v3/calendars/REMOVED/events?maxResults=200&orderBy=startTime&showDeleted=true&singleEvents=true&timeMax=2015-02-16T20%3A13%3A03-08%3A00&timeMin=2014-10-17T20%3A13%3A03-07%3A00&updatedMin=2014-11-19T16%3A05%3A42-08%3A00

And we received the 410 error above.  So that was 5 minutes after we had previously "synced" that time range, and it failed.  I know an event was added to that calendar during those 5 minutes.

Any help would be appreciated as this seems to be impacting quite a few of our calendars now.

Thanks!

[ds]

This is creating big problems for us too.

One thing is to recommend using the new synctoken method (which was added only recently, it wasn't there when we upgraded to v3 by the way).

Another thing is to reduce the updatedMin limit arbitrarily and silently, without giving prior advice to anyone. You gave developers ample time to upgrade API from v2 and v3, and that worked well. But now you change an IMPORTANT parameter in your limits, without giving prior notice.

Did you not realize that you were going to break the functionality of many clients using the API with this change? Who is responsible for this decision?

updatedMin is still listed as supported in your v3 docs. You recommend the new syncToken, but we can also keep using updatedMin according to your docs. So don't change updatedMin limits without giving prior notice to anyone, and then tell us that it's our fault that we're not using synctokens.

[Frank Cort]

The nextSyncToken feature is very cool.  However, the docs say (and I just confirmed) it doesn't work if you are specifying timeMin or timeMax when fetching events.   Which means, in order to use it, you have to sync all of a calendar's events.  For some calendars that could be a huge number of events.  We typically only store a range of events to reduce overhead on our side, API requests, and privacy liability.

Does anyone know if the nextSyncToken feature can work with a time range of events?  Thanks!

[Alexey Panteleev]

We have the same situation, we use timeMin/timeMax and we have not switched to using syncTokens yet.

It seems Google started rolling out a code update through their servers where they disabled the usual (20 days cut off) support for updateMin parameter.

This is so bad for us, we see more and more of our users get stuck with the error. How about backward compatibility, Google?

Is this a bug on Google side or a decision to turn off this option?

[Alexey Panteleev]

Are we supposed to do a full reload after getting the 410 error? Would that allow continuing to use updateMin? Will try that.

[Gaurav Nolkha]

We are running into this issue too. Could this be related to this thread:

http://stackoverflow.com/questions/7244246/generate-an-rfc-3339-timestamp-similar-to-google-tasks-api

[Yannick Biet]

Do you have same 410 error when switching to syncTokens ?

[Jakub Zavodny]

Hi all, thanks for reporting this.

In some cases we do respond with 410 unnecessarily, we are working on fixing it.

[Frank Cort]

FYI, after more experimenting, I found that syncTokens do work when specifying timeMin and timeMax now.  So the docs might be out of date.  It wasn't working for me because I was still including the "orderBy" parameter.  Once I removed that parameter, I started getting syncTokens.  

But, consistency issues aside, I still think using UpdatedMin is a better option for us because it allows us to have a sliding time window that we keep in sync while avoiding large initial syncs (as long as 410s are not being returned erroneously).  i.e. when UpdatedMin is working, we can do one initial sync of a time range from X to Y (where X and Y are dates).  Then each day we advance X and Y by one day.  When we do the advance, we only have to do a full initial sync to fetch the events on the Y + 1 day.  The X+1 through Y can still be updated using the UpdatedMin.  

Unless I'm missing something, the syncToken approach doesn't allow for a sliding window like this.  When we advance by 1 day we'd have to do a full initial sync again.   Alternatively we could have two Sync Tokens and do a "Google Maps Tile" style of fetching events.  But with that, we'd have to do a full initial sync every Y-X days.  

Anyhow, I'm very relieved to see that Google is addressing the UpdatedMin/410 issues.  Thanks!

On Wednesday, November 19, 2014 5:06:41 PM UTC-8, Frank Cort wrote:

[ds]

Frank,

we have exactly the same approach, i.e. an initial sync of 365 days ahead, and then every day we do a full sync of one day (365 days away) to keep always the next 365 days synced. And then we do periodic syncs with updatedMin to stay up to date with new events/changes/deletions.The updateMin approach works perfectly for this, we've never missed an event. It is not clear by Google's documentation if an app can have multiple sync tokens in place for the same refresh token, i.e. does doing a full sync of one day invalidate the nextSyncToken received from the last sync query that used a syncToken? Or doing a new full sync simply returns a new nextSyncToken but the previous syncToken can still be used?

Either way, if this reduction in limits (from 20 days to few hours) was intentional, Google should have given advance notice to let developers prepare. If this (as it now seems) is an unintended bug, it would be nice if Google developers posted a note here once the bug is fixed.

[Alexey Panteleev]

Yep, we essentially have the same approach too. Yesterday we figured out a workaround for this, it seems.

We do a full (in our terms) realod without using the updatedMin parameter once when we encounter 410 and then we can continue using the incremental calls with updatedMin again.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Calendar API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-calendar-api/CTJgaiuTu-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-calendar...@googlegroups.com.
To post to this group, send email to google-ca...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-calendar-api.
For more options, visit https://groups.google.com/d/optout.

[Lucia Fedorova]

Hi all, I would like to clarify two things:

From Frank: But, consistency issues aside, I still think using UpdatedMin is a better option for us because it allows us to have a sliding time window that we keep in sync while avoiding large initial syncs (as long as 410s are not being returned erroneously).  i.e. when UpdatedMin is working, we can do one initial sync of a time range from X to Y (where X and Y are dates).  Then each day we advance X and Y by one day.  When we do the advance, we only have to do a full initial sync to fetch the events on the Y + 1 day.  The X+1 through Y can still be updated using the UpdatedMin.  

This one breaks when an event gets moved outside of the currently synced window. Imagine you did a sync for a window between Nov 1 and Nov 31. There was an event on Nov 20. Now this event got moved to Dec 3rd. As your updatedMin request has constraints you will not know that the event is gone and you will keep displaying it (which makes your user think there is a meeting even though there is none).

From Frank: Unless I'm missing something, the syncToken approach doesn't allow for a sliding window like this.  When we advance by 1 day we'd have to do a full initial sync again.   Alternatively we could have two Sync Tokens and do a "Google Maps Tile" style of fetching events.  But with that, we'd have to do a full initial sync every Y-X days. 

As you say, you can always move the window by doing a full sync for the new window and then continue syncing using sync tokens. Even though you will possibly get a little more data on sync using the sync tokens, you can post filter the data and you are guaranteed to not send your users to meetings which are no longer taking place.

To unsubscribe from this group and all its topics, send an email to google-calendar-api+unsub...@googlegroups.com.
To post to this group, send email to google-calendar-api@googlegroups.com.

[ds]

Lucia,

when you say:

"As your updatedMin request has constraints you will not know that the event is gone"

that is not correct as long as the app calls google to list events with the updatedMin parameter with a regular frequency. If you set updatedMin to be the time of your last call (or slightly earlier), and make a new call before the elapsed time exceeds the updatedMin limit, then the call succeeds because it's within the limits, and you don't miss out on any change or deletion - if google honours the request according to the docs. The key point is that if updatedMin is too far in the past, Google will reply with a 410 error, and so our apps WILL know that something went wrong and a new full sync is needed.

as to your second answer:

if our app does a full sync (no updatedMin) of the next 365 days, it receives a nextSynctoken, say ABC. If it later does a 'window' full sync of days 366 and 367, it will receive a second nextSynctoken, say DEF. Will ABC still be valid as syncToken though? It is not clear from the API's documentation. The point is that us (and, it seems, many others posting in this thread), have one ongoing 'updated only' sync, executed periodically, and one 'full' sync, on a future date range, executed periodically too. Would the latter break the synctokens of the first?

To post to this group, send email to google-ca...@googlegroups.com.

[Lucia Fedorova]

that is not correct as long as the app calls google to list events with the updatedMin parameter with a regular frequency. If you set updatedMin to be the time of your last call (or slightly earlier), and make a new call before the elapsed time exceeds the updatedMin limit, then the call succeeds because it's within the limits, and you don't miss out on any change or deletion - if google honours the request according to the docs. The key point is that if updatedMin is too far in the past, Google will reply with a 410 error, and so our apps WILL know that something went wrong and a new full sync is needed.

I don't think you understood what I meant. The event did not disappear because of deletion. There was an update operation performed which moved this event outside the window. Feel free to try it out.

as to your second answer:

if our app does a full sync (no updatedMin) of the next 365 days, it receives a nextSynctoken, say ABC. If it later does a 'window' full sync of days 366 and 367, it will receive a second nextSynctoken, say DEF. Will ABC still be valid as syncToken though? It is not clear from the API's documentation. The point is that us (and, it seems, many others posting in this thread), have one ongoing 'updated only' sync, executed periodically, and one 'full' sync, on a future date range, executed periodically too. Would the latter break the synctokens of the first?

You should always use the latest sync token, thus DEF in this case. The reason is that ABC could be too old (that's why you needed the full sync) and then trying to use it again will only result in 410 repeatedly.

[Frank Cort]

Hi Lucia,

Thanks for the clarifications.

I see what you mean about the event moving out of the window... but in practice that hasn't been an issue for us.  In practice, appointments are only moved a few days ahead in time if moved at all, and the sliding window catches up before anything bad happens. The only time it might be a real issue is if the user moved the event forward 6 months or more, but that rarely happens from what I've seen. But yes, you're right, the sliding window approach using UpdatedMin has that pit fall, and if you're trying to stay 100% in sync, that's a huge pit fall.

So, could you provide us with a Sliding Window using SyncTokens?  The process would be:

1. Client does initial sync by fetching events with timeMin=X and timeMax=Y and stores sync token.

2. Client does incremental event fetches periodically (or on-demand through push/watch notifications) throughout the day using sync token only.

3. After 24 hours, client does event fetch with new timeMin=X+1 day, new timeMax=Y+1 day, and existing sync token.  Server returns events that changed since the sync token provided in the intersection between the time ranges PLUS all the events in the new time range that weren't in the old time range.  A new sync token could be returned and stored by the client.

4. repeat steps 2 and 3.

I think I might just be suggesting that the server does the "post filtering" you mentioned instead of the client.  That would reduce API requests and bandwidth usage.   

I suspect a large number of applications that sync with Google Calendar attempt to do some sliding window approach, so having a way to do that without requiring occasional large full syncs (except when 410 is returned) would be beneficial to everyone including Google, right?

Thanks again!

Also, are the fixes in place for the extra UpdatedMin 410s (i.e. the original issue here)?  We're seeing that they've dropped back to normal levels.  

---------------------------------------------------------------

Frank Cort

Support Specialist, Apptoto.com

Email: fr...@apptoto.com

Phone: 888-318-3765 ext. 701

To unsubscribe from this group and all its topics, send an email to google-calendar...@googlegroups.com.
To post to this group, send email to google-ca...@googlegroups.com.

[ds]

Luicia,

1. when we do the updatedMin sync, we do not use a window of dates, we ask for anything that has changed since updatedMin, regardless of its date in the calendar. the window is the other sync that is done regularly to grab events that were already there, but beyond the initial 365 days sync

2. how long before ABC synctoken becomes old? as soon as DEF is returned? Keep in mind that we regularly do a sync with the ABC synctoken (or its successor, obviously). It would not be more than a few hours from the time the synctoken was received.

The key question is whether as soon as DEV is generated ABC becomes invalid immediately, or if ABC expires only due to age.