Objectify's new memcache integration: The good, the bad

[Jeff Schnitzer]

The new code for memcache is complete. The good:

* Should not go out of sync even under heavy contention (only a
DeadlineExceededException can break it)
* Usable outside of Objectify
* Collects statistics and even has a primitive servlet to render them
* The code is a lot cleaner

The bad:

* I still can't make Ofy work gracefully (ie, allow uninstall()) with
the RemoteApiInstaller. There are three separate issues Google's
issue tracker, any of which fixed would solve this particular problem.

* Because there are no batch CAS operations on MemcacheService, batch
operations in Objectify are now painfully slow. A get() of 50
entities is at best 50 separate memcache requests, each of which will
take 4-5ms. Cache misses are even more expensive - 3X more with a
totally cold cache. While memcache will reduce datastore operations,
this really destroys a lot of the value - for large batches, latency
will be significantly higher than a full trip to the datastore even
for cache hits.

My new favorite issue in the GAE tracker is this one:

http://code.google.com/p/googleappengine/issues/detail?id=4874

If I could wave a magic wand and fix any of them, this would be it.
Ok, maybe SSL first. But this is definitely #2. Memcache needs batch
versions of getIdentifiable()/putIfUntouched(), or this feature will
have questionable value. I'm debating whether it's even a good idea
to release a 3.1beta with the new cache.

Jeff

[Jon Stevens]

Hi all,

Please notice that this email was sent at 4:36am.

I've been sitting next to Jeff for the last two days and he's been
working non-stop on this functionality for all of us.

Really good software engineers make it look easy.

cheers,

jon

[Tom Carchrae]

I had exactly the same thought this morning.  Kudos Jeff.  I'd really like to tell you to stop it and get back to your startup, but I'm incredibly grateful for your dedication.  No doubt this pays off in many ways.

Is there anything the rest of us can do aside from be thankful?  I've already starred all the blocker issues, and I had a quick poke through the code this morning but was unclear of where/how to help.  

Out of curiousity, would other cache providers be easier to work with?  Or would it ever make sense to run your own backend as a memcache server?   http://stackoverflow.com/questions/731738/java-memcached-client http://code.google.com/p/spymemcached/wiki/Optimizations

Tom

[Ronoaldo José de Lana Pereira]

Many thanks Jeff and Jon, sincerely.

As Tom said, how can we help you guys?

[Jeff Schnitzer]

On Wed, Sep 28, 2011 at 1:22 PM, Tom Carchrae <t...@carchrae.net> wrote:
> I had exactly the same thought this morning.  Kudos Jeff.  I'd really like
> to tell you to stop it and get back to your startup, but I'm
> incredibly grateful for your dedication.  No doubt this pays off in many
> ways.

I rationalize that this is on the critical path to launch because we
make heavy use of caching and the new app has much more rigid
transactional requirements than my previous apps. But it was also an
itch that needed to be scratched. Don't tell Jon :-)

> Is there anything the rest of us can do aside from be thankful?  I've
> already starred all the blocker issues, and I had a quick poke through the
> code this morning but was unclear of where/how to help.

Beg/bribe/extort anyone you know at Google into implementing batch CAS
operations on memcache. There's over 400 people on this mailing list,
if they all starred this issue it would rise to #17 on the list:

http://code.google.com/p/googleappengine/issues/detail?id=4874

If you're reading this, please star :-)

I have high hopes for this one because as far as I can tell from the
memcached protocol, it should be easy to implement - especially batch
getIdentifiable().

> Out of curiousity, would other cache providers be easier to work with?  Or
> would it ever make sense to run your own backend as a memcache server?
>   http://stackoverflow.com/questions/731738/java-memcached-client http://code.google.com/p/spymemcached/wiki/Optimizations

Running your own memcached-analogue as a backend is certainly
possible, but I suspect fairly undesirable:

* Large RAM backends are insanely expensive - like an order of
magnitude too expensive.

* You have to talk to them through URLFetch and HTTP. Depending on
how Google has optimized this, there will likely be several network
hops involved (app instance to fetch service, fetch service to
loadbalancer, loadbalancer to backend instance). Each adds
undesirable latency that isn't present in the (presumably) app
instance -> memcached scenario.

* I'm willing to bet that Jetty is at least two full orders of
magnitude slower than memcached at processing requests. The nature of
memcached is that 90% of the cost is processing the protocol; a
fetch/store is almost a no-op by comparison. Everything about
memcached has been designed for this; everything about Java appservers
has been designed for database hits, page assembly, etc.

Regarding those links, you can't use sockets on GAE so you can't
support the memcached protocol directly as either client or server.

Jeff

[Ikai Lan]

Jeff, I could be mistaken (someone can show me numbers to show otherwise), but URLFetches from a front end instance to a backend should be relatively low latency. I'm not saying it'll be faster than Memcache, but it probably doesn't make as many network hops as the worst case scenario you've mentioned (I don't think there is a load balancer involved).

You are probably right about Jetty being not fast enough, but I'm not in a position to say, having not benchmarked it myself. There are definitely situations where using a backend as a cache may be desired, but a backend full on substitute for all the things Memcache can do is probably not desirable.

--

Ikai

[Jeff Schnitzer]

On Wed, Sep 28, 2011 at 2:05 PM, Ikai Lan <ik...@google.com> wrote:
> Jeff, I could be mistaken (someone can show me numbers to show otherwise),
> but URLFetches from a front end instance to a backend should be relatively
> low latency. I'm not saying it'll be faster than Memcache, but it probably
> doesn't make as many network hops as the worst case scenario you've
> mentioned (I don't think there is a load balancer involved).

Ah, that would make sense if you hit the full backend address to a
specific replica instead of the general backend name.

> You are probably right about Jetty being not fast enough, but I'm not in a
> position to say, having not benchmarked it myself. There are definitely
> situations where using a backend as a cache may be desired, but a backend
> full on substitute for all the things Memcache can do is probably not
> desirable.

Yeah, I wasn't suggesting that backends can't be useful - backends
give you (mostly) predictable expiration and critical section
control... but they're a pretty poor substitute for memcached
especially considering that memcached on appengine is more or less
free. Just needs a batch CAS...

Really, it just needs a batch getIdentifiable(). I'm pretty ok with
cache misses being expensive. Cache hits need to be fast.

Jeff

[Jeff Schnitzer]

Proper documentation is now here:

http://code.google.com/p/objectify-appengine/wiki/MemcacheStandalone

I'd really like to hear from anyone using it under load. The next
release of Objectify will be 3.1 beta 1... after I get a little more
comfortable with it. I tried out the new cache in Similarity, but
because Similarity makes really heavy use of batch fetches (50+ at a
time) it was too slow and I had to revert. Without big batch fetches
it should be fine, though.

Jeff

On Wed, Sep 28, 2011 at 4:36 AM, Jeff Schnitzer <je...@infohazard.org> wrote:

[Stefano Ciccarelli]

I've just returned from holidays, in the next few days I will give a try to the beta release.

Thanks

Stefano