map lock - replacing data - hz thread never returns

[gise...@yachay.net]

I have a tomcat application, and currently started just one node, multicast mode
I am populating a cache and proceeded to lock the map, but when there is data to put, the application never returns since it is trying to repartition, but there is ONLY one node and no data existed before.

If I do not lock the map, then it works.
ICache is a wrapper on the IMap. Most ICache methods are the same as IMap's.  ICache.lock(long, TimeUnit) calls IMap.lockMap(long, TimeUnit)
This is the code
    public void replaceSectionSlots(final Map inReplacement) {
        ICache<String, Set<Positionable>> sectionSlotCache = getYardSectionSlotCache();
        //  It should not be null.
        if (sectionSlotCache != null) {
            boolean locked = false;
            try {
                locked = sectionSlotCache.lock(500, TimeUnit.MILLISECONDS);
                if (locked) {
                    sectionSlotCache.removeAll();                       //  There is no data; it never returns
                    sectionSlotCache.putAll(inReplacement);
                    //  Section Slot observers should be notified
                    LOGGER.debug("replaceSectionSlots");
                }
            } catch (Throwable thr) {
                LOGGER.error("Could not replace all blocks. Exception: " + thr.getMessage());
            } finally {
                if (locked) {
                    sectionSlotCache.unlock();
                }
            }
        }
    }


//// This is the log
2012-05-02 16:02:21,449 DEBUG  [hz.navis-instance.ServiceThread    ] [FactoryImpl:809] (   ) /134.37.94.111:5701 [navis_dev] Instance created ProxyKey {name='c:dynamicViewsCache-9-blockSlot', key=null}
2012-05-02 16:03:00,539 DEBUG  [hz.navis-instance.scheduled.thread-1] [CMap:91] (   ) /134.37.94.111:5701 [navis_dev] c:dynamicViewsCache-9-blockSlot Cleanup , dirty:0, purge:0, evict:0, unknown:0, stillOwned:0, backupPurge:0
2012-05-02 16:03:00,519 DEBUG  [hz.navis-instance.scheduled.thread-2] [PartitionManager:?] (   ) /134.37.94.111:5701 [navis_dev] Checking partition table for repartitioning...
2012-05-02 16:03:03,680 DEBUG  [hz.navis-instance.cached.thread-6  ] [ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev] CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
2012-05-02 16:03:04,181 DEBUG  [hz.navis-instance.cached.thread-6  ] [ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev] CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
2012-05-02 16:03:04,681 DEBUG  [hz.navis-instance.cached.thread-6  ] [ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev] CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
2012-05-02 16:03:05,181 DEBUG  [hz.navis-instance.cached.thread-6  ] [ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev] CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
....

[Mehmet Dogan]

Hazelcast is not trying to repartition. It is just checking partition table if there is a need for repartitioning.

What is implementation of ICahce.removeAll()? Does it just call IMap.clear()? 

What is your Hazelcast version? IMap.clear does not call CONCURRENT_MAP_SET operation but there are several CONCURRENT_MAP_SET calls in log. Are there other threads processing on this map? 

@mmdogan

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/7IjQGnFKbEsJ.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[gise...@yachay.net]

I am using version 2.0.3.
ICache.removeAll calls IMap.clear()

    public void removeAll() {
        _cache.clear();
    }

There are other threads waiting for a caller of replaceSectionSlots() to finish, but it never comes back.  I see the finally never gets called.
At the point that this happens, the cache is empty, so clear() should not have found anything.

To unsubscribe from this group, send email to hazelcast+unsubscribe@googlegroups.com.

[Mehmet Dogan]

Can you post thread dump of all threads when removeAll call blocks?

@mmdogan

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/PxYO8Lp-Q0QJ.

To post to this group, send email to haze...@googlegroups.com.

To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.

[gise...@yachay.net]

It is the putAll() that is blocking, sorry.  There was no data. So it is trying to add a map with data.
I am attaching two files. One with all the thread dumps. The other with some of the threads, end of log, and code snippets with commented line numbers.

The one with all the threads, defaultQuartzScheduler_Worker-15 and ActiveMQ Task-2 threads are waiting for another process with is not used by this application.
The first thread,defaultQuartzScheduler_Worker-2, is where hazelcast, never returns, and that piece of code is part of a LOCKed block executed only when the application starts, and only one thread is allowed to enter
at a time.

[Mehmet Dogan]

Oh, how I didn't realize! IMap.putAll() uses executor threads for put operations, that optimizes puts due to target nodes. And because map is locked by caller thread, executor threads processing putAll() sub-tasks are blocked. Locking the whole map and doing putAll is cause of deadlock.

Actually locking whole map (without key) is not a preferred way, because it actually freezes map and blocks all other operations. Maybe you should use a separate lock for that task.

@mmdogan

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/8Aw9SZbJ6EwJ.

To post to this group, send email to haze...@googlegroups.com.

To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.

[gise...@yachay.net]

So, what are we allowed to do when we lock the whole cache? so that we do not run into these deadlocks?

[Mehmet Dogan]

You should avoid using IMap.lockMap(), as said before it just freezes map. Single target operations are safe to use with locking the whole map. Multi-target operations are executed through a thread-pool and they will be blocked because of lock.

@mmdogan

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/bBp_PCvua-sJ.

To post to this group, send email to haze...@googlegroups.com.

To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.