Indexes and near cache?

[Joe Planisky‏]

We just tried adding indexes to some of our maps (using Hazelcast 2.1.2) to improve query performance. We're running into some unexpected behavior. Before we dive into it, I thought I'd ask if this is a known issue or maybe even expected.

We're seeing what looks like stale data being returned in query results. We have one map in particular with a small number of items, each of which are updated once every 5 seconds. We also run a query on that map every 5 seconds. After several minutes of apparently correct operation (i.e. returning the most recent version of items from the map), the query seems to get "stuck" returning a stale item even though an updated copy of that item has been "put" into the map several times. Once it starts returning this stale item, it never seems to get an updated copy. It's almost as if a near cache is being used with invalidation turned off even though we do not have near cache configured for this map.

Does adding indexes to a map automatically enable near caching? If not, and there's no obvious explanation for this, we'll try to come up with a test case.

--
Joe

[Mehmet Dogan‏]

No, indexing does not enable near cache. Do you see the updated value when you run map.get(key)?

@mmdogan

--
Joe

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Joe Planisky‏]

We first saw this problem in Hazelcast 2.1.2. We thought upgrading to Hazelcast 2.4 might have fixed this issue since it did have some fixes related to queries, but alas, it's still with us. Reduced to it's simplest statement: doing a predicate query on an IMap that has indexes sometimes returns stale data, while retrieving a value with IMap.get(key) always returns the most current value.

I've attached a program that demonstrates the problem. The attached zip file contains a hazelcast.xml configuration file, the demo program itself (Demo.java), and a Bash shell script to drive part of the demo. Because this problem seems to happen most often in the first few queries made to the map, the shell script repeatedly cycles the demo. The bug happens very rarely, so the cycle script may have to run for many minutes or even hours to catch an incident.

Demo has a client and a server mode. The client mode does nothing except provide nodes to let Hazelcast distribute the map data to. The server mode adds some objects to an IMap, and then enters a loop where it:

1 retrieves one of the objects (with IMap.get(key))
2 updates a value in the object
3 puts the modified object back into the map (with IMap.put(key, value))
4 waits a random amount of time (< 100 ms)
5 then retrieves the object with both a call to IMap.get(key) and a query
6 compares the values from both retrieval methods
7 if the values are different, print an error message
8 loop back to step 1, repeat for 5 seconds.

The modified values should always match since there's only one "server" updating them. However, very rarely (but often enough to cause us great grief), they don't match. Once a mismatch occurs, it is never healed (i.e. all subsequent queries will return the same stale value.)

Compile Demo.java with something like

$ javac -classpath .:<path-to-hazelcast-jar> Demo.java

Start several copies of Demo in client mode (I usually start 3 copies in separate terminal windows) with

$ java -cp .:<path-to-hazelcast-jar> Demo

Edit cycle.sh, replacing <path-to-hazelcast-jar> with the path to your hazelcast-all.jar file. Then start the ./cycle.sh shell script which will repeatedly run the Demo in server mode until it detects an error. This may take minutes to hours, but I usually get the error within about 30 minutes on my Mac OS X system.

Hope this helps you find the problem.

--
Joe

[Joe Planisky‏]

I've come up with a much easier and more reliable way to reproduce this issue of map queries returning stale data. Simply run multiple (up to 6) copies of the attached Demo program with the attached config file. If you manage to get 6 copies running without seeing the error, try repeatedly killing and restarting one of the instances. I usually see the error by the time I start the 4th instance, but sometimes I can get all 6 running.

The problem seems to occur when a new node joins the cluster while a query is in process.

--
Joe

[Joe Planisky‏]

I have opened issue #359 for this problem.

https://github.com/hazelcast/hazelcast/issues/359

--
Joe