major MEMCACHE changes started October 5th

[James Gilliam]

GAE has drastically changed the way MEMCACHE works; as a result my application (ogeekcom) overall usage jumped by approximately 5 times with the same approximate bandwidth output.  Like a 400% increase in price.

Specifically, they are purging shared memcache very aggressively -- possible in an effort to force people to signup for paid memcache.

As a result of this change, my application is using many more datastore reads and many more instances to compensate for the poor memcache performance.

Like always, this was done without any announcement at all.

If they made this change to increase make applications cost more to run, it is illegal.  

There is no problem with them offering a premium service for memcache, but it is illegal to degrade the previous service to force people into the paid model.

[Kaan Soral]

Out of pure curiosity, how is it illegal? :)

I always check my app after seeing instance/memcached etc issues, the performance/usage has been steady for 2 years

on what kind of traffic/usage have you seen 400% increase?

[vlad]

That is nasty news though not illegal. Let's see what gae says in response.

[Vinny P]

On Sun, Oct 6, 2013 at 1:57 PM, James Gilliam <jimgi...@gmail.com> wrote:

As a result of this change, my application is using many more datastore reads and many more instances to compensate for the poor memcache performance.

GAE production recently transitioned from 1.8.4 to 1.8.5, and there's always a bit of instability when these transitions occur. Did these problems start recently, or is it a chronic issue? 

 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

[timh]

Can't say I have noticed any significant difference in memcache other than the usual week or so of instability when they roll out new releases.

Hit count:	70713
Miss count:	5933
Hit ratio:	92%
Item count:	1979 item(s)
Total cache size:	40909470 byte(s)
Oldest item age:	23 hour(s) 36 min(s) 40 second(s)

T

[Kaan Soral]

yey :) didn't even know these stats existed :)

[James Gilliam]

Memcache continues to perform poorly; I suspect they are making many changes to it.

Saturday was awful -- aggressively purging memcache, Sunday was better, Monday was worse than Sunday.

At this point, not sure what to do ... are we dealing with just code instability or some fundamental change?

Really wish GAE would test these changes before deploying them.

Or at least alert the community to whatever changes they are making.

[R.J. Lorimer]

Just saw this in the Pre-release documentation for 1.8.6 - this seems like it will be helpful to you moving forward. Work for this change (upgrading the memcache backends, etc) may also have contributed to the amount of flushing you've seen:

- A memcache size chart has been added to admin console's dashboard.  Access it

  via the drop-down above the graph.  The chart graphs memcache size over time

  enabling customers to determine when cache flush events occurred. This is a

  preview feature.

Unfortunately, memcache is inherently unreliable (by design) in GAE - they shut it off all the time when data center maintenance is going on. The above graph will at least help with visibility on how memcache performs.

I do think it would be helpful for small shops to have a lower priced dedicated/guaranteed tier for memcache. The 100/month is going to be more than small shops who pay for a one, two, or even three reserved instances pay for the actual instances.

Right now I suspect that Google is considering the shared service good enough for these smaller hosted apps, but I can imagine having a more "stable" 50-250MB memcache instance would be hugely useful for a lot of apps.

[Kristopher Giesing]

Are you sure this is something Google is doing, and not something another memcache client in your same data center is doing?

A more charitable explanation for all this is that memcache performance is hard to guarantee without explicit allocations (i.e. the paid option), and that you've been lucky up until recently.

- Kris

[timh]

I would check the totals of how much you are storing in memcache.  There seems to be a high water mark (never seen it documented) which once you exceed 

there is a lot of churn.  I have noted in the past that memcache can be affected locally.  (ie one instances contents of memcache gets purged a lot, whilst another

see's long retention times, which means there has to be localized effect in the infrastructure.)

On Monday, October 7, 2013 2:57:02 AM UTC+8, James Gilliam wrote:

[Jim Gilliam]

I am using about 30 megs of memory cache for 1200 items -- not very much imo.  In the past I have used as much as 100 megs without seeing excessive purges.

Hard to code against parameters which are not documented and possibly even change based on unknown factors.

At this point, you can even assume that a memcache will be around for even a few minutes before being purged -- if it is purged that often why not just deny the memcache.set in the first place.

Further, it is not even clear what you get with the premium memcache; I suspect the per gigabyte refers to the memcache bandwidth and not the amount of cache being saved. 

Whenever GAE changes something fundamental, they don't tell us about it and we have to try and figure out what they might have changed.  Not cool imo.

My guess is they penalize people not using the premium memcache which is just another way of them forcing people to move to the premium version and making the non premium version worthless.

I am probably looking at a serious recoding effort to reduce my reliance on memcache.

Not happy about any of this.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/JC4xSIJRV7I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[Vinny P]

On Tue, Oct 8, 2013 at 12:24 PM, James Gilliam <jimgi...@gmail.com> wrote:

Memcache continues to perform poorly; I suspect they are making many changes to it.

Saturday was awful -- aggressively purging memcache, Sunday was better, Monday was worse than Sunday.

At this point, not sure what to do ... are we dealing with just code instability or some fundamental change?

Can you try changing the instance class size to a larger class ( ie from F1 to F2 ) and then back again? Sometimes applications get stuck with "damaged" instances and experience reduced reliability. Changing the instance class size forces the scheduler to reallocate new instances, which will hopefully open on a more reliable machine/different place within the physical Google infrastructure.

[Jacob Taylor]

From my experience using dedicated memcache it is billed by GB stored. You are limited in transactions by GB stored as well. We have been using one GB for a little while and have not had any cache drops. I have loved being able to rely on it more than I could when it was shared. Shared cage is based on load, frequency of reads, and other people's usage. We had good success with it but quickly found out that short term rate limiting by tokens were not sticking around long enough. Now we have the space to keep the locks around for days.

We are putting millions of items in the cache though.

Jacob

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

[Alex Burgel]

I'm experiencing the same thing as you. A few months of reliable memcache, but now since the weekend I'm seeing the cache reset every couple of hours and a lower hit rate.

Nothing changed in my app (haven't deployed in a few days).

[Jim Gilliam]

Alex,

Basically, GAE had a price increase by making shared memcache unusable.  You have to signup for the premium memcache at 12 cents a GB-hour to get back to the way it use to work.

This was dishonest of GAE and they have not even announced that they essentially deprecated the shared memcache since it is no longer useable for much of anything.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/JC4xSIJRV7I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[James Gilliam]

The minimum private memcache costs 12 cents a GB-hour, and 1 GB is the minimum you can specify.  You pay for the entire gigabyte, even if you are only using 1/10 of it.  So, it costs $2.88 a day, nearly $90 a month; instead of 30 cents a day.   And it was free before Saturday.

This is ridiculous.  It should be prorated and/or let you specify a much lower amount of memcache, like 100 Megs.

This is a massive price increase which is forced on people because of the changes to shared memcached on Saturday.

Bottom line -- GAE cannot be trusted.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:

[timh]

I don't believe the evidence you present can necessarily lead one to the conclusion that google is actively purging memcache to force you to buy premium memcache.

If they where doing such a thing I would imagine lots of people would be seeing the same behaviour as you.

T

On Monday, October 7, 2013 2:57:02 AM UTC+8, James Gilliam wrote:

[James Gilliam]

Here is what I know so far ...

I ran the premium memcache for 18 hours yesterday and saw no cache purging.  My cache size grew to 110 megs with 6000 items and the oldest cache item was 18 hours.    Since the minimum cache is 1 gig, I assume there would be no purging until the size was above that.

When using shared cached, the purging is now very aggressive.  Since Saturday, I have seen the oldest cache item at less than 15 mins, total cache of 10 megs and items at 500 or less.  It does vary a lot but as the purging increases the cost of the application increases, as does the performance.  Right now it is performing better than these numbers but with the shared cache you never know when they will purge the shit out of the cache and the application starts consuming other resources to compensate for poor memcache performance.

Something clearly changed on Saturday when I saw my costs increase by 4 to 5 times as a direct result of memory cache purging. 

The only way you can guarantee some amount of memory cache is with the premium memcache product and the minimum size is 1 gig (roughly 10 times what I really need).  This costs 12 cents per hour per gig, 2.88 per day, roughly 90 per month, over 1000 per year.

It would be less painful if you could allocate smallest than 1 gig, e.g. 100 megs.  So the costs would drop by a factor of 10.

I will be forced to opt for the premium memcache if the shared cache continues to perform so poorly, or rewrite lots of code so I am less dependent on memcache.  Both of these are direct burdens from GAE changing shared memcache.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:

GAE has drastically changed the way MEMCACHE works; as a result my application (ogeekcom) overall usage jumped by approximately 5 timesd.  Like a 400% increase in price.

[Kaan Soral]

I received an email a while ago from a google sales rep about dedicated memcached, s/he couldn't sell it to me, until your post I didn't even gave memcached too much thought although I have a pre-ndb custom implementation of a memcached layer, but you certainly sold dedicated memcached to me :)

I'm also curious about what kind of improvement dedicated memcached will provide, I might post results here if I detect a significant change, my memcached was probably also purged more frequently than usual, the age was always around ~1 hour, but since it's a cache I didn't care, until now

[R.J. Lorimer]

James, have you tried adjusting your instance class up and then back down as Vinny P suggested above? Have your instances rotated out since this started?

I'm not personally surprised that the premium memcache performed better for you; it's certainly supposed to.

As for myself (like others posting here), I haven't seen a degradation in the performance of the shared memcache over the past several weeks, so I don't think this is some sort of global scheme by Google to force us all to upgrade to dedicated memcache. Is it possible last saturday that your instance rotated out? Maybe you got stuck with an instance that is coping with a particularly disruptive memcache installation.

I can understand why you're frustrated, but I don't think that assuming malice on the part of Google is the most constructive action. I believe that something changed on Saturday in your memcache experience, but it doesn't have to be an underhanded move by Google. More than likely, it's just a quirk or glitch in the instance fabric as Vinny has suggested could be the case. It's possible that you are now in a part of the data center with more aggressive neighbors. The very nature of shared memcache is such that there are no guarantees that other apps in your same local region of the data center won't impact your performance.

[Alex Burgel]

I filed a production issue on this. I suggest everyone who is seeing this problem do the same.

I also switched the dedicated memcache service until this is fixed. so far so good.

[James Gilliam]

Now there is a billing issue with memcache premium.  It goes along with 0.00 all day and then plunk all of it at once.  It doesn't add the cost hour by hour -- clearly a bug. The next day, they plunk like 18 hours right at the beginning of the day, which is rolled back later in the day.  But they show the WRONG billing all through the day and I am afraid they will deactivate my application with these errors.

In my opinion, they don't do any real beta testing before plunking stuff out.  My guess is they let developers push whenever they want, at least that is what is suggested with the types of error they create.  Saturday is the day the love pushing out bad code.

They should make premium memcache free until they fix the problems and let us opt in for less than 1 gig --- 1 gig of memcache is way TOO much for many apps and they have made free shared memcache worthless -- their plan all along imo.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:

[Alex Burgel]

I see this bug as well. Dedicated memcache is in preview release, so bugs like this shouldn't be surprising.

I filed an issue:

https://code.google.com/p/googleappengine/issues/detail?id=10108

[pdknsk]

Saturday is the day the love pushing out bad code.

 I laughed when I read this. This should be the subtitle of the (largely useless) App Engine Status page, in bold.

[James Gilliam]

There should be a GAE Legal Defense Fund to represent people when GAE acts inappropriately.  I am totally dependent on GAE for my business and I suspect many other people are as well.  What GAE has done with MEMCACHE is wrong imo and we need somebody to call them out on this formally.  What they did was a wholesale price increase by reducing the function of shared memcache, literally forcing people like me to opt for the paid premium version.  The operative word here is "forcing" which is coercion in my view.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:

[Barry Hunter]

I'm still intrigued how you know this is deliberate?

Granted you've had a bad experience, but it does not mean there is a conspiracy against you.  

Have you tried redeploying, will often end up in a different part of the infrastructure, and so move out of a neighbourhood with a noisy neighbour. Might have to repeat the process a few times to see the effect. 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

[James Gilliam]

Barry --

I am not arguing a conspiracy at all.  I am saying that what GAE did, whether it be on purpose or not, resulted in a huge increase in the cost of hosting.  It has forced me to opt in for premium memcache.  It remains to be seen if GAE responds to this at all but the history suggests they will not.  Also, they changed this on Saturday without any warning either -- something they should be held accountable for as well.

Right now,  I have to opt in for the premium memcache at a cost of almost $1100 a year because 1 gig is the minimum size I can get.  They do not even prorate the cost based on what I am actually using -- something they routinely do in other billing areas, e.g. reads, bandwidth, emails, etc.   So, why not prorate the premium hosting.  In my case the cost could drop by 90% since I will likely not use more than 100 megs.

[James Gilliam]

GAE just responded to the billing issue with the following

Updates:

        Status: WorkAsIntended

Comment #1 on issue 10108 by jbelmo...@google.com: dedicated memcache billed all at once

http://code.google.com/p/googleappengine/issues/detail?id=10108

The billing table in the dashboard reflects the accounting done to manage your app's daily spend.  For resources billed in units of time like dedicated memcache, for the purposes of daily spend the assumption is made that you'll have the resource enabled for the remainder of the day.  Hence 17.17 hours reflects the time from when you enabled dedicated memcache until midnight.

Your actual dedicated memcache usage will be reflected correctly in the daily billing usage report on the usage history page.

Once again, GAE missed the point.

By inaccurately showing MEMCACHE billing during the day and declaring works as intended, the app is at risk of being stopped when the billing is reached.

I had to up my daily billing because of their bug.

While we are on inaccurate real time billing ... they also do not add the bandwidth charges during the day ... another bug they will undoubtedly call works as intended.

[Barry Hunter]

On 11 October 2013 09:42, James Gilliam <jimgi...@gmail.com> wrote:

Barry --

I am not arguing a conspiracy at all.  I am saying that what GAE did, whether it be on purpose or not, resulted in a huge increase in the cost of hosting.

How do you know something was 'changed'? The evience presented so far does still point being something like a noisey neighbour problem. As noted by others, not everyone is experiencing the 'issue'

 

 It has forced me to opt in for premium memcache.

Have you tried any of the ideas presented in the thread? Redeploying etc. 

There's also CloudSQL, could use that as a cache. FOr your data sounds like about $130 a year. 

 

 It remains to be seen if GAE responds to this at all but the history suggests they will not.  

I also dount they will bother, with all this rhetoric 

 

Also, they changed this on Saturday without any warning either -- something they should be held accountable for as well.

Right now,  I have to opt in for the premium memcache at a cost of almost $1100 a year because 1 gig is the minimum size I can get.  They do not even prorate the cost based on what I am actually using -- something they routinely do in other billing areas, e.g. reads, bandwidth, emails, etc.   So, why not prorate the premium hosting.  In my case the cost could drop by 90% since I will likely not use more than 100 megs.

On Sunday, October 6, 2013 11:57:02 AM UTC-7, James Gilliam wrote:

GAE has drastically changed the way MEMCACHE works; as a result my application (ogeekcom) overall usage jumped by approximately 5 times with the same approximate bandwidth output.  Like a 400% increase in price.

Specifically, they are purging shared memcache very aggressively -- possible in an effort to force people to signup for paid memcache.

As a result of this change, my application is using many more datastore reads and many more instances to compensate for the poor memcache performance.

Like always, this was done without any announcement at all.

If they made this change to increase make applications cost more to run, it is illegal.  

There is no problem with them offering a premium service for memcache, but it is illegal to degrade the previous service to force people into the paid model.

[James Gilliam]

I try and/or evaluate all the suggestions.  The worse day for shared memcache was Saturday. Today and yesterday was much improved.  However, they still more aggressively purge shared memcache than they use to.  I know this because I track it everyday.  The premium memcache is great function since it allows more cache; however 1 gig is too much for me and too costly ... $1100 a year.  Not reporting the real time billing correctly is a pain but something I can live with.  No idea if they depreciated the shared memcache on purpose to force me into premium, but whatever they did screwed me -- that is for sure.

[timh]

I really think you confuse various localised and/or widespread problems with intent.  I just don't see "intent"

I have been using appengine (totally reliant on it) since 2008.  memcache has always been vague in it's rentention of records.  See me earlier response to this thread , you see I had oldest items sitting at 23 hours on the 7th yesterday when I looked it was 2 hours.  Right now it's sitting at 12 mins.   It varies all the time.  Go back 3 years, rentention times where all over the shop.

[Vinny P]

On Fri, Oct 11, 2013 at 1:02 PM, James Gilliam <jimgi...@gmail.com> wrote:

I try and/or evaluate all the suggestions.  The worse day for shared memcache was Saturday. Today and yesterday was much improved.  However, they still more aggressively purge shared memcache than they use to.  I know this because I track it everyday.  The premium memcache is great function since it allows more cache; however 1 gig is too much for me and too costly ... $1100 a year.  Not reporting the real time billing correctly is a pain but something I can live with.

Hello James,

Are you experiencing this behavior across multiple application IDs, or is it one application having performance issues? What you can try is - if you have a paid support package - is to call Google up and ask them to move your app ID to a different place in the physical infrastructure. You might get better neighbors sharing your server, and less contention for memcache resources. If you don't have a paid support contract, you can try redeploying your app as Barry noted above.

In addition (depending on your application's size and instance class) you may be able to cram some data into an in-instance cache. Try profiling your app and checking to see if there's some free RAM to use.

With that said, I +1 Tim and Barry's comments, this just doesn't look intentional on Google's part.