java.util.concurrenttimeoutexception for some client

[Arghya Kusum Das]

I want to upload 32GB of Data with almost 800 million keys to Hazelast and using 16 machines with 32GB RAM per machine. Several Hazelcast client is uploading the data to the DHT simultaneously.

Suddenly, I started getting the concurrent timeout exception for some client. Can any one tell me what is the problem and how to fix it?

[Peter Veentjer]

Can you post the stacktrace?

On Tue, Feb 10, 2015 at 6:30 AM, Arghya Kusum Das <arghyakus...@gmail.com> wrote:

I want to upload 32GB of Data with almost 800 million keys to Hazelast and using 16 machines with 32GB RAM per machine. Several Hazelcast client is uploading the data to the DHT simultaneously.

Suddenly, I started getting the concurrent timeout exception for some client. Can any one tell me what is the problem and how to fix it?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/6d0033ab-2c7b-48fc-b49d-03b0314e1892%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rahul]

This is not the first case of Concurrent TimeoutException reported. See the below stack trace in 3.4 on simple ops such as put, replace and remove:

Exception in thread "main" com.hazelcast.core.HazelcastException: java.util.concurrent.TimeoutException

at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:74)

at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrConnect(ClientConnectionManagerImpl.java:337)

at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.tryToConnect(ClientConnectionManagerImpl.java:273)

at com.hazelcast.client.spi.impl.ClientInvocationServiceImpl.invokeOnKeyOwner(ClientInvocationServiceImpl.java:81)

at com.hazelcast.client.spi.impl.ClientInvocationServiceImpl.invokeOnKeyOwner(ClientInvocationServiceImpl.java:71)

at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:126)

at com.hazelcast.client.proxy.ClientMapProxy.remove(ClientMapProxy.java:209)

at TestApp.main(TestApp.java:343)

Caused by: java.util.concurrent.TimeoutException

at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)

at java.util.concurrent.FutureTask.get(FutureTask.java:119)

at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrConnect(ClientConnectionManagerImpl.java:334)

... 6 more

[Peter Veentjer]

@Rahul:

This is caused by the client not being able to make a connection to a remote system.

@Arghya

Can you post your stacktrace.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/437e303a-9c5b-4f80-a517-c789599816ea%40googlegroups.com.

[Arghya Kusum Das]

Hi,

Thanks for your quick reply.

I ran it on Supercomputer. And Unfortunately I forgot to keep the log.

I will run it again tomorrow and post the stacktrace.

Thanks and Regards,

Arghya Kusum Das

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/BfBJM3Xf0Jg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGuAWdAC%2BaniDx84DriJWxwEStJ%3D_ZqZoGzUA%2Bz%3D-Bc7xZtCJw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks and regards,

Arghya Kusum Das

(225-270-6163)

[Arghya Kusum Das]

In the meantime, if you have any quick fix (Since it is well known issue) please forward thst to me I will include that too and let you know the result.

[Arghya Kusum Das]

Hi,

The stacktrace is as follows:

om.hazelcast.core.HazelcastException: java.util.concurrent.TimeoutException
	at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:74)
	at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrConnect(ClientConnectionManagerImpl.java:337)
	at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.tryToConnect(ClientConnectionManagerImpl.java:273)
	at com.hazelcast.client.spi.impl.ClientInvocationServiceImpl.invokeOnKeyOwner(ClientInvocationServiceImpl.java:81)
	at com.hazelcast.client.spi.impl.ClientInvocationServiceImpl.invokeOnKeyOwner(ClientInvocationServiceImpl.java:71)
	at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:126)
	at com.hazelcast.client.proxy.ClientMapProxy.put(ClientMapProxy.java:352)
	at com.hazelcast.client.proxy.ClientMapProxy.put(ClientMapProxy.java:200)
	at GraphLoader.load(GraphLoader.java:85)
	at LoadGraphToDHT$GraphToDHTMap.map(LoadGraphToDHT.java:43)
	at LoadGraphToDHT$GraphToDHTMap.map(LoadGraphToDHT.java:1)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
	at java.util.concurrent.FutureTask.get(FutureTask.java:119)
	at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrConnect(ClientConnectionManagerImpl.java:334)
	... 17 more

Size of map when it failed: 

85 597 899 (eighty-five million five hundred ninety-seven thousand eight hundred ninety-nine)

[M. Sancar Koyunlu]

I believe this is fixed in 3.4.1

See https://github.com/hazelcast/hazelcast/pull/4300/

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEp5LX4aui-EFmJ27aXUfe%3D0AzxT3Go-p4AStJGrsm3vw6X56w%40mail.gmail.com.

[Arghya Kusum Das]

Hi,

Thaks for your reply. 

Actually I started using Hazelcast from this Jan. I amusing the 3.4 jar. I never tried to build Hazelcast from source. In fact I don't have the source code. 

Is there any binary (jar) available for 3.4.1. Or, is there any documentation how to build the code.

Thanks and Regards,

Arghya Kusum Das

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEj0St9_HSybQ9LKK16pTFtO%2BXTBnk9MDmRMoXCXdiSmYEdeng%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[M. Sancar Koyunlu]

We are in the release stage right now for 3.4.1 . It will be available in probably next week after our stability tests passes. 

Building from source should be straightforward since it has no dependency and it is a maven project. 

See

https://github.com/hazelcast/hazelcast#building-from-source

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEp5LX6D%3DpPB70OWf%2Bf_g-QB8_%3D016Fv0Lnzv3Ok_8GSA42jQw%40mail.gmail.com.

[M. Sancar Koyunlu]

And I forget to mention, for 3.4.1 you need to build from 3.4.1 branch. This branch will be tagged as v3.4.1 and removed after release.  

master branch is currently  3.5-SNAPSHOT.

maintenance-3.x branch is currently 3.4.2-SNAPSHOT.

[Arghya Kusum Das]

Hi,

I tested with 3.4.1 and yes it is fixed.

But I got another problem. Hazelcast is taking too much of memory than expected.

My data size is 31GB. Number of unique keys are 642Million.

I am running Hazelcast on 32 machines each with 32GB RAM.

I ran only one hazescast server instance per machine with 24GB Heap Space.

Surprisingly, After loading all the data, the memory usage profile is showing almost 15GB memory usage per machine (average). 

That is, the total memory usage for my 31GB data is 15*32=480GB !!!!

Which does not make any sense. Is it some configuration issue? Am I missing something? Or, is a bug?

Thanks and Regards,

Arghya Kusum Das

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEj0St8otuPhyqr4Bsqw91n-fN933mCqR6uMZC95xT1ttC4mDQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.