Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
map lock - replacing data - hz thread never returns
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  8 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
gisel...@yachay.net  
View profile  
 More options May 2 2012, 7:38 pm
From: gisel...@yachay.net
Date: Wed, 2 May 2012 16:38:09 -0700 (PDT)
Local: Wed, May 2 2012 7:38 pm
Subject: map lock - replacing data - hz thread never returns

I have a tomcat application, and currently started just one node, multicast
mode
I am populating a cache and proceeded to lock the map, but when there is
data to put, the application never returns since it is trying to
repartition, but there is ONLY one node and no data existed before.

If I do not lock the map, then it works.
ICache is a wrapper on the IMap. Most ICache methods are the same as
IMap's.  ICache.lock(long, TimeUnit) calls IMap.lockMap(long, TimeUnit)
This is the code
    public void replaceSectionSlots(final Map inReplacement) {
        ICache<String, Set<Positionable>> sectionSlotCache =
getYardSectionSlotCache();
        //  It should not be null.
        if (sectionSlotCache != null) {
            boolean locked = false;
            try {
                locked = sectionSlotCache.lock(500, TimeUnit.MILLISECONDS);
                if (locked) {
                    sectionSlotCache.removeAll();                       //  
There is no data; it never returns
                    sectionSlotCache.putAll(inReplacement);
                    //  Section Slot observers should be notified
                    LOGGER.debug("replaceSectionSlots");
                }
            } catch (Throwable thr) {
                LOGGER.error("Could not replace all blocks. Exception: " +
thr.getMessage());
            } finally {
                if (locked) {
                    sectionSlotCache.unlock();
                }
            }
        }
    }

//// This is the log
2012-05-02 16:02:21,449 DEBUG  [hz.navis-instance.ServiceThread    ]
[FactoryImpl:809] (   ) /134.37.94.111:5701 [navis_dev] Instance created
ProxyKey {name='c:dynamicViewsCache-9-blockSlot', key=null}
2012-05-02 16:03:00,539 DEBUG  [hz.navis-instance.scheduled.thread-1]
[CMap:91] (   ) /134.37.94.111:5701 [navis_dev]
c:dynamicViewsCache-9-blockSlot Cleanup , dirty:0, purge:0, evict:0,
unknown:0, stillOwned:0, backupPurge:0
2012-05-02 16:03:00,519 DEBUG  [hz.navis-instance.scheduled.thread-2]
[PartitionManager:?] (   ) /134.37.94.111:5701 [navis_dev] Checking
partition table for repartitioning...
2012-05-02 16:03:03,680 DEBUG  [hz.navis-instance.cached.thread-6  ]
[ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev]
CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
2012-05-02 16:03:04,181 DEBUG  [hz.navis-instance.cached.thread-6  ]
[ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev]
CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
2012-05-02 16:03:04,681 DEBUG  [hz.navis-instance.cached.thread-6  ]
[ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev]
CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
2012-05-02 16:03:05,181 DEBUG  [hz.navis-instance.cached.thread-6  ]
[ConcurrentMapManager:?] (   ) /134.37.94.111:5701 [navis_dev]
CONCURRENT_MAP_SET BeforeRedo target Address[134.37.94.111:5701]
....


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mehmet Dogan  
View profile  
 More options May 3 2012, 1:59 am
From: Mehmet Dogan <meh...@hazelcast.com>
Date: Thu, 3 May 2012 08:59:38 +0300
Local: Thurs, May 3 2012 1:59 am
Subject: Re: map lock - replacing data - hz thread never returns

Hazelcast is not *trying to repartition**. *It is just checking partition
table if there is a need for repartitioning.

What is implementation of ICahce.removeAll()? Does it just call
IMap.clear()?

What is your Hazelcast version? IMap.clear does not call CONCURRENT_MAP_SET
operation but there are several CONCURRENT_MAP_SET calls in log. Are there
other threads processing on this map?

@mmdogan


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gisel...@yachay.net  
View profile  
 More options May 3 2012, 12:24 pm
From: gisel...@yachay.net
Date: Thu, 3 May 2012 09:24:45 -0700 (PDT)
Local: Thurs, May 3 2012 12:24 pm
Subject: Re: map lock - replacing data - hz thread never returns

I am using version 2.0.3.
ICache.removeAll calls IMap.clear()

    public void removeAll() {
        _cache.clear();
    }

There are other threads waiting for a caller of replaceSectionSlots() to
finish, but it never comes back.  I see the finally never gets called.
At the point that this happens, the cache is empty, so clear() should not
have found anything.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mehmet Dogan  
View profile  
 More options May 4 2012, 9:04 am
From: Mehmet Dogan <meh...@hazelcast.com>
Date: Fri, 4 May 2012 16:04:11 +0300
Local: Fri, May 4 2012 9:04 am
Subject: Re: map lock - replacing data - hz thread never returns

Can you post thread dump of all threads when removeAll call blocks?

@mmdogan


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gisel...@yachay.net  
View profile  
 More options May 4 2012, 7:25 pm
From: gisel...@yachay.net
Date: Fri, 4 May 2012 16:25:44 -0700 (PDT)
Local: Fri, May 4 2012 7:25 pm
Subject: Re: map lock - replacing data - hz thread never returns

It is the putAll() that is blocking, sorry.  There was no data. So it is
trying to add a map with data.
I am attaching two files. One with all the thread dumps. The other with
some of the threads, end of log, and code snippets with commented line
numbers.

The one with all the threads, defaultQuartzScheduler_Worker-15 and ActiveMQ
Task-2 threads are waiting for another process with is not used by this
application.
The first thread,defaultQuartzScheduler_Worker-2, is where hazelcast, never
returns, and that piece of code is part of a LOCKed block executed only
when the application starts, and only one thread is allowed to enter
at a time.

  deadlock_info.txt
17K Download

  full_thread_dump.txt
105K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mehmet Dogan  
View profile  
 More options May 5 2012, 4:12 am
From: Mehmet Dogan <meh...@hazelcast.com>
Date: Sat, 5 May 2012 11:12:57 +0300
Local: Sat, May 5 2012 4:12 am
Subject: Re: map lock - replacing data - hz thread never returns

Oh, how I didn't realize! IMap.putAll() uses executor threads for put
operations, that optimizes puts due to target nodes. And because map is
locked by caller thread, executor threads processing putAll() sub-tasks are
blocked. Locking the whole map and doing putAll is cause of deadlock.

Actually locking whole map (without key) is not a preferred way, because it
actually freezes map and blocks all other operations. Maybe you should use
a separate lock for that task.

@mmdogan


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gisel...@yachay.net  
View profile  
 More options May 7 2012, 2:11 pm
From: gisel...@yachay.net
Date: Mon, 7 May 2012 11:11:32 -0700 (PDT)
Local: Mon, May 7 2012 2:11 pm
Subject: Re: map lock - replacing data - hz thread never returns

So, what are we allowed to do when we lock the whole cache? so that we do
not run into these deadlocks?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mehmet Dogan  
View profile  
 More options May 8 2012, 2:20 am
From: Mehmet Dogan <meh...@hazelcast.com>
Date: Tue, 8 May 2012 09:20:03 +0300
Local: Tues, May 8 2012 2:20 am
Subject: Re: map lock - replacing data - hz thread never returns

You should avoid using IMap.lockMap(), as said before it just freezes map.
Single target operations are safe to use with locking the whole map.
Multi-target operations are executed through a thread-pool and they will be
blocked because of lock.

@mmdogan


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic