Memcache problem on a production application

[Mathieu Clavel]

Hello,

My production application is currently having memcache problem.

Nothing is visible in the global appengine system status : https://code.google.com/status/appengine/

Error is :

Caused by: com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (...)

Nothing is working : Jasper also has error :

org.apache.jasper.runtime.JspFactoryImpl internalGetPageContext: Exception initializing page context
com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (_ahsXuelLhc5ClPxy8Dra2vPUQ)

My application name is oav-agz.

Regards,

Mathieu

[Mathieu Clavel]

Memcache is now working as expected.

Problem was from 16h to 18h (UTC+2)

Mathieu

[Per]

Hi Mathieu,

not sure if you're aware of this (it took me a while as well) but you need to design your application so it doesn't flinch even when memcache is unavailable. Try-catch everything. We had an issue at the same time as you as well, and it turned out that in one part of the application we had also forgotten to try-catch memcache errors. Just keep improving until it's 100% robust. Because memcache just isn't. A large scale downtime would probably show up on the system status page, but local issues won't.

Cheers,
Per

[jon]

This is actually a GAE bug. According to GAE documentation the
memcache service should by default log and ignore infrastructure
faults (which would be a very sensible thing to do) but it doesn't.

I opened a ticket 6 months ago (yes, half a year!)
http://code.google.com/p/googleappengine/issues/detail?id=6236

There is no excuse for leaving this broken for so long. It's an
extremely harmful bug and yet easy to fix. I've been raising this
issue every chance I get but the response from fellow GAE developers
on this group has been muted. We should be outraged, people :-)
Seriously, this bug can take down any of your sites if it happens to
link up with a dodgy memcache service. From experience this can last
up to half an hour.

[Michael Hermus]

I am not quite outraged, but I did star the issue :)

It looks like the status is 'started', so that is a good sign...

[Mathieu Clavel]

Hi Per,

yes my app is not design to work with memcache failure.

It's also not designe for other appengine failure but that's another story.

Even if I did design or if I redesign it to be memcache failure resilient, appengine is also using it for jsp (see the logs in my first message), so : no jsp... Too bad.

I'm not going to have other problem with appengine as my app is already rewritten to run on a Tomcat server.

I'm still using gwt, but bye-bye appengine and datastore, hello tomcat and mysql.

Regards,

Mathieu

[Mathieu Clavel]

Hi Jon,

I also starred your ticket, but as I'm leaving appengine, I'm not going to have this problem anymore.

Others problems on Tomcat, sure, but I will be in control of the application server.

Thanks,

Mathieu

[Rick Mangi]

Wise man.

[jon]

Thanks for starring Mathieu. I you don't mind, why are you leaving?

Michael the ticket has been stuck on started for a long while now :-(

[Mathieu Clavel]

The application is used in as a production.

Appengine was driving to develop specific implementations using tasks and/or backends to bypass HardDeadlineExceed Exceptions.

We have no control on the app server : maintenance, what to do when there are problems, how many time berfore it starts working again, ...

More specifically, we are going to use webservices necessitating client certificat and also to integrate the application to SAP.

Mathieu

[Sridhar Ragunathan]

HI,

I was trying to access it using a backend.

will that also give HardDeadlineExceed Exceptions

tks 

Sridhar r

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/PX3yQrxAtokJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.