Cache consistency & distributed cache

[Satish]

Now that I have used memcache as a second level cache I'm going to
analyze a few things on the cache that's been set up. I would
appreciate if anyone can give me inputs on this.

1) Cache consistency
2) Distributed cache.

How does memcache instance handle these?

Thanks,
Satish

[Ray Krueger]

> Now that I have used memcache as a second level cache I'm going to
> analyze a few things on the cache that's been set up. I would
> appreciate if anyone can give me inputs on this.
>
> 1) Cache consistency
> 2) Distributed cache.

Your best bet at this point is to read the memcached faq at
http://www.socialtext.net/memcached/index.cgi?faq
Especially the "How does memcached work" section...
http://www.socialtext.net/memcached/index.cgi?faq#how_does_memcached_work

Also if you want to have a more in depth conversation about memcached
itself chat with the whole group at
http://groups.google.com/group/memcached?pli=1

This list is really just for the hibernate-memcached list. I can
gladly answer your questions, but the real experts are over there :)

Memcached is distributed in the sense that the client hashes keys to
decide which memcached node to put an object in. When you are only
running one instance (with a client configured to talk to one
instance) you don't get that effect. The key is to run many instances.
The client will hash and then mod the key by the number of servers to
decide where to get/set from. If one of your memcached instances goes
down, you only lose the data that was stored in that instance.

Hope that helps!

[Ronak Patel]

Ray,

 

I noticed one thing though...if I only have one memcached server and I have something cached there, I get an odd behaviour when memcached goes down.

 

Instead of Hibernate reaching out the database, it constantly tries to hit memcached. Why is this the cache? Is there a way to configure hibernate+memcached to fail over to the database whenever no memcached servers are available?

[Ray Krueger]

> I noticed one thing though...if I only have one memcached server and I have
> something cached there, I get an odd behaviour when memcached goes down.
>
> Instead of Hibernate reaching out the database, it constantly tries to hit
> memcached. Why is this the cache? Is there a way to configure
> hibernate+memcached to fail over to the database whenever no memcached
> servers are available?

Hibernate-memcached doesn't know anything about your database. When
the memcached client can't get to the server it is going to return
nulls to Hibernate. When Hibernate gets a null back from the cache it
goes to the database.
Hibernate always asks the cache before going to the database, that's
the point of having a cache enabled. Your going to see log messages
about spymemcache trying to connect, and even hibernate-memcached
trying to do lookups when the memcached isntances are down as there's
no reason for it to stop trying.

Be sure that running one instance of memcached in development is fine,
but when you go to production you're defeating the purpose of using
memcached if you're only running one instance.

[Ronak Patel]

This is the point...I don't see Hibernate going to the database.

It just hangs there pinging the cache again and again in an endless loop...

Maybe I have a bug somewhere??

[Ray Krueger]

> This is the point...I don't see Hibernate going to the database.
> It just hangs there pinging the cache again and again in an endless loop...
> Maybe I have a bug somewhere??

Sorry, I mean that when you request data from hibernate, it will go
the database when the cache returns null. In the meantime, spymemcahed
will be checking the server to see if it can reconnect.

[Ronak Patel]

I will double check this behaivor 2night. I remember it polling memcached but I don't remember seeing any data coming back from Hibernate even if the database was there.

[Ronak Patel]

Ray,

 

That is the exact behaviour I get...it looks like Hibernate is just hitting memcached forever and the amount of time it takes for Hibernate to respond to the query directly from the database takes forever but does return.

 

I am returning about 60 records from the database in this query and I find it quite problematic that it takes Hibernate a couple of minutes to return the data even though the L2 cache server is down.

 

Would you happen to know why this is the case? Is this the default behaviour or is there a way to configure hibernate to not take this long?

[Ray Krueger]

> That is the exact behaviour I get...it looks like Hibernate is just hitting
> memcached forever and the amount of time it takes for Hibernate to respond
> to the query directly from the database takes forever but does return.
>
> I am returning about 60 records from the database in this query and I find
> it quite problematic that it takes Hibernate a couple of minutes to return
> the data even though the L2 cache server is down.
>
> Would you happen to know why this is the case? Is this the default behaviour
> or is there a way to configure hibernate to not take this long?
>

Well that sucks :P

Out of curiosity are you setting an "operationTimeout" value at all?
The default is like 1 second :)

Would it be possible for you to try the latest batch of code at Github out?
http://github.com/raykrueger/hibernate-memcached/
If added better exception handling that might help here when there are
NO memcached instances available. As I've said before, you definitely
shouldn't be running one instance alone though.
If they're all down it still shouldn't increase your query time.

You can either clone it with git or just download it by clicking the
download link. That will zip or tar the latest code for you. From
there you can just build it with Maven (mvn install).
Unfortunately The pom.xml I pulled over from SVN still has "1.0" as
the version, you should change that to 1.1-SNAPSHOT or something.

Let me know if that changes the behavior at all. Thanks!

[Ray Krueger]

Actually, sorry, is your query just slow? Is that what you were saying? :)

[Ronak Patel]

It is fast when memcached is present but has no objects cached.

It is also fast when memcached is not even used (I have no L2 cache set).

 

The query slows down when I set memcached as the L2 cache, the cache is populated, but all of the memcached servers are down.

 

It seems to be a tcp/ip timeout that slows things down. Seems like you iterate through the whole resultset and try to fetch data from the cache one at a time...

[Ray Krueger]

On Mon, Nov 24, 2008 at 11:09 AM, Ronak Patel <rona...@gmail.com> wrote:
> It is fast when memcached is present but has no objects cached.
> It is also fast when memcached is not even used (I have no L2 cache set).
>
> The query slows down when I set memcached as the L2 cache, the cache is
> populated, but all of the memcached servers are down.
>
> It seems to be a tcp/ip timeout that slows things down. Seems like you
> iterate through the whole resultset and try to fetch data from the cache one
> at a time...
>

Hibernate controls what data goes in and out of the cache and how the
lookups are handled. I have no control over that. All I do is map
those requests into memcached.
You might want to pull down the spymemcached source and see if you can
see something worth fixing in there. Right now though, I haven't seen
this behavior or heard of anyone else having a similar problem. I'll
see if I can dig something up as to what sort of timeout it's using.

[RaviKumar Potnuru]

Hi Ray, Patel,

Do you have a solution for this problem are still not fixed, in my case I am getting a response but hibernate-memcached is taking decision after almost 60 seconds. :-(