Getting com.hazelcast.transaction.TransactionException: Transaction is timed-out! in hazelcast 3.1.2

[rites...@gaikai.com]

Still digging through the errors in hazelcast with our load test on 2 node cluster over the weekend.

We are seeing this error in log:


com.hazelcast.transaction.TransactionException: Transaction is timed-out!
        at com.hazelcast.transaction.impl.TransactionImpl.checkTimeout(TransactionImpl.java:255)
        at com.hazelcast.transaction.impl.TransactionImpl.commit(TransactionImpl.java:227)
        at com.hazelcast.transaction.impl.TransactionContextImpl.commitTransaction(TransactionContextImpl.java:59)
        at com.hazelcast.client.txn.CommitTransactionRequest.call(CommitTransactionRequest.java:38)
        at com.hazelcast.client.CallableClientRequest.process(CallableClientRequest.java:30)
        at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.run(ClientEngineImpl.java:368)
        at com.hazelcast.util.executor.ManagedExecutorService$Worker.run(ManagedExecutorService.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
        at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)
        at ------ End remote and begin local stack-trace ------.(Unknown Source)
        at com.hazelcast.client.util.ErrorHandler.returnResultOrThrowException(ErrorHandler.java:33)
        at com.hazelcast.client.spi.impl.ClientClusterServiceImpl.sendAndReceiveFixedConnection(ClientClusterServiceImpl.java:208)
        at com.hazelcast.client.txn.TransactionProxy.sendAndReceive(TransactionProxy.java:148)
        at com.hazelcast.client.txn.TransactionProxy.commit(TransactionProxy.java:95)
        at com.hazelcast.client.txn.TransactionContextProxy.commitTransaction(TransactionContextProxy.java:64)
        at com.gaikai.alert.AlertUtil.runInTransaction(AlertUtil.java:104)
        at com.gaikai.alert.processor.impl.AlertSender.run(AlertSender.java:159)
        at java.lang.Thread.run(Thread.java:744)

We have 2 concurrent clients which are reading jobs from same queue and processing items from queue. We are using transactional queues to to get the items.

It seems tx commit has timedout, in our case in a transaction we are doing:

(1) poll the queue and get an id (String)  (tx queue)
(2) do a lookup on a map using this id and find another id (non tx lookup)
(3) do a few lookup and map updates in tx maps
(4) add id to a different queue (tx queue)

Is having a non transactional lookup like in step (2) can be an issue? What would cause the time out and is there way to prevent it? Whats the best way to handle this (we are already retrying after backoff)

Thanks
Ritesh