Unable to connect terracotta cluster server
370 views
Skip to first unread message
harikrishna b
Jan 19, 2017, 11:28:20 AM1/19/17
to terracotta-oss
- What version of Terracotta Server you are currently using; TC server 5
- Paste the configuration you are using;
- Providing JDK and OS versions maybe useful as well. Jdk 8 and windows
client tc-config.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tc-config xmlns="http://www.terracotta.org/config">
<servers secure="false">
<server host="localhost" name="testing" bind="0.0.0.0">
<logs>C:\Users\terracotta\server-logs</logs>
<tsa-port bind="0.0.0.0">9510</tsa-port>
<management-port bind="0.0.0.0">9540</management-port>
<tsa-group-port bind="0.0.0.0">9530</tsa-group-port>
</server>
<client-reconnect-window>120</client-reconnect-window>
</servers>
</tc-config>
=======================
server logs
------------------------------
2017-01-09 23:40:29,855 [main] INFO com.tc.server.TCServer - Server started as testing
2017-01-09 23:40:34,861 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.ElectionManagerImpl - Election Complete : [Enrollment [ NodeID[localhost:9510], isNew = true, weights = 0,0,116,-6093981732416992524 ]] : State[ Election-Complete ]
2017-01-09 23:40:34,861 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.StateManagerImpl - Becoming State[ ACTIVE-COORDINATOR ]
2017-01-09 23:40:34,861 [WorkerThread(l2_election_handler_stage, 0)] INFO com.terracottatech.console - Becoming State[ ACTIVE-COORDINATOR ]
2017-01-09 23:40:34,861 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - putToStore StripeID[7a1c76996ff24e19ab669d112b468b96]
2017-01-09 23:40:34,861 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - Notify local StripeID ready
2017-01-09 23:40:34,876 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.ElectionManagerImpl - Declared as Winner: Winner is : Enrollment [ NodeID[localhost:9510], isNew = true, weights = 0,0,116,-6093981732416992524 ]
2017-01-09 23:40:34,876 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Starting TSA services...
2017-01-09 23:40:34,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.terracottatech.console - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.
2017-01-09 23:40:35,292 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCConnection - Comms Message Batching enabled
2017-01-09 23:40:35,346 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@737495433: connected: true, closed: false local=127.0.0.1:9510 remote=127.0.0.1:56803 connect=[Mon Jan 09 23:40:35 EST 2017] idle=32ms [0 read, 0 write]
2017-01-09 23:40:35,346 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - "Client Cannot Reconnect. Connection attempts from the Terracotta node at 127.0.0.1:56803 are being rejected by the Terracotta server array. Reason: Stack for ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[] not found."
2017-01-09 23:40:35,361 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[]: CLOSE EVENT : com.tc.net.core.TCConnectionImpl@737495433: connected: false, closed: true local=127.0.0.1:9510 remote=127.0.0.1:56803 connect=[Mon Jan 09 23:40:35 EST 2017] idle=0ms [152 read, 495 write]. STATUS : START
2017-01-09 23:40:35,361 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[]: closing down connection - com.tc.net.core.TCConnectionImpl@737495433: connected: false, closed: true local=127.0.0.1:9510 remote=127.0.0.1:56803 connect=[Mon Jan 09 23:40:35 EST 2017] idle=0ms [152 read, 495 write]
2017-01-09 23:41:50,092 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1651296138: connected: true, closed: false local=127.0.0.1:9510 remote=127.0.0.1:56826 connect=[Mon Jan 09 23:41:50 EST 2017] idle=2ms [0 read, 0 write]
2017-01-09 23:41:50,113 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - Allocated connection license for jvm a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a; ConnectionPolicy[maxConnections=2147483647, connectedJvmCount=1]
2017-01-09 23:41:50,114 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - New connection [0] from jvm a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a
2017-01-09 23:41:50,114 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - User null successfully authenticated
2017-01-09 23:41:50,115 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCWorkerCommManager - Selecting [L2_L1:TCWorkerComm # 0, FD, wt:0] from [[L2_L1:TCWorkerComm # 0, FD, wt:0], [L2_L1:TCWorkerComm # 1, FD, wt:0], [L2_L1:TCWorkerComm # 2, FD, wt:0], [L2_L1:TCWorkerComm # 3, FD, wt:0]]
2017-01-09 23:41:50,116 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@1280936924
2017-01-09 23:41:50,128 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TSA Server - Health monitoring agent started for 127.0.0.1:56826
2017-01-09 23:41:50,135 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TSA Server - HealthCheckCallbackPort verification PASSED for 127.0.0.1:56826(callbackport: 56825)
2017-01-09 23:41:50,154 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Handling client handshake for ClientID[0]
2017-01-09 23:41:50,154 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Sending handshake acknowledgement to ClientID[0]
2017-01-09 23:41:50,246 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.objectserver.entity.ManagedEntityImpl - Client:ClientID[0] Invoking lifecycle CREATE_ENTITY on EntityID{className='org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient', entityName='VoltronReadWriteLock-EhcacheClientEntityFactory-AccessLock-testing'}
2017-01-09 23:41:50,269 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.objectserver.entity.ManagedEntityImpl - Client:ClientID[0] Invoking lifecycle FETCH_ENTITY on EntityID{className='org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient', entityName='VoltronReadWriteLock-EhcacheClientEntityFactory-AccessLock-testing'}
2017-01-09 23:41:50,386 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.objectserver.entity.ManagedEntityImpl - Client:ClientID[0] Invoking lifecycle CREATE_ENTITY on EntityID{className='org.ehcache.clustered.client.internal.EhcacheClientEntity', entityName='testing'}
2017-01-09 23:41:50,413 [WorkerThread(request_processor_stage, 2, 4)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(request_processor_stage, 2, 4),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
java.lang.AssertionError: Server failed to retrieve EhcacheStateService.
at org.ehcache.clustered.server.EhcacheActiveEntity.<init>(EhcacheActiveEntity.java:181)
at org.ehcache.clustered.server.EhcacheServerEntityService.createActiveEntity(EhcacheServerEntityService.java:50)
at org.ehcache.clustered.server.EhcacheServerEntityService.createActiveEntity(EhcacheServerEntityService.java:32)
at com.tc.objectserver.entity.ManagedEntityImpl.createEntity(ManagedEntityImpl.java:601)
at com.tc.objectserver.entity.ManagedEntityImpl.invokeLifecycleOperation(ManagedEntityImpl.java:365)
at com.tc.objectserver.entity.ManagedEntityImpl.lambda$processLifecycleEntity$61(ManagedEntityImpl.java:224)
at com.tc.objectserver.entity.ManagedEntityImpl$$Lambda$26/1625390904.run(Unknown Source)
at com.tc.objectserver.entity.ManagedEntityImpl$SchedulingRunnable.run(ManagedEntityImpl.java:958)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.invoke(RequestProcessor.java:161)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.run(RequestProcessor.java:154)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:27)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:23)
at com.tc.async.impl.StageQueueImpl$FlushingHandledContext.runWithHandler(StageQueueImpl.java:527)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:197)
2017-01-09 23:41:50,428 [WorkerThread(request_processor_stage, 2, 4)] ERROR com.tc.logging.ThreadDumpHandler - Mon Jan 09 23:41:50 EST 2017
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):
"Dump On Timeout Timer" Id=65 TIMED_WAITING on java.util.TaskQueue@5f9b9a75
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"WorkerThread(client_handshake_stage, 0)" Id=64 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a250ab7
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(passive_replication_ack_stage, 0)" Id=63 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a9b6f96
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(respond_to_request_stage, 0)" Id=62 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30c2e550
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(voltron_message_stage, 0)" Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3309890f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(active_to_passive_driver_stage, 0)" Id=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11db4d6b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"Static Member discovery" Id=58 WAITING on com.tc.net.groups.TCGroupMemberDiscoveryStatic@301d6719
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.tc.net.groups.TCGroupMemberDiscoveryStatic.pauseDiscovery(TCGroupMemberDiscoveryStatic.java:277)
at com.tc.net.groups.TCGroupMemberDiscoveryStatic$1.run(TCGroupMemberDiscoveryStatic.java:182)
at java.lang.Thread.run(Thread.java:745)
"WorkerThread(l2_state_message_handler_stage, 0)" Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1752fc2f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 3, 4)" Id=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@686c4e3b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 2, 4)" Id=55 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
at com.tc.util.runtime.ThreadDumpUtil.getThreadDump(ThreadDumpUtil.java:143)
at com.tc.logging.ThreadDumpHandler.callbackOnExit(ThreadDumpHandler.java:41)
at com.tc.lang.ThrowableHandlerImpl.handleDefaultException(ThrowableHandlerImpl.java:211)
- locked <0x2397e648> (a java.lang.Object)
at com.tc.lang.ThrowableHandlerImpl.handleThrowable(ThrowableHandlerImpl.java:147)
at com.tc.lang.TCThreadGroup.uncaughtException(TCThreadGroup.java:38)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1952)
"WorkerThread(request_processor_stage, 1, 4)" Id=54 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78e9688c
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 0, 4)" Id=53 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78da503a
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_discovery_stage, 0)" Id=52 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72cdc48c
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_handshake_message_stage, 0)" Id=51 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ef4d93b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(receive_group_message_stage, 0)" Id=50 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d6a7161
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(server_entity_response_message_stage, 0)" Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@771ba176
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(l2_state_change_stage, 0)" Id=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b6ee2bd
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(l2_election_handler_stage, 0)" Id=47 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@239bc3ea
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_events_dispatch_stage, 0)" Id=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d745f81
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 3, 4)" Id=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ce5b48
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 2, 4)" Id=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b293a06
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 1, 4)" Id=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66502e1a
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 0, 4)" Id=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6957ed06
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_hydrate_message_stage, 0)" Id=41 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1cb6ef7f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(platform_information_request, 0)" Id=40 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4e1d0fe5
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"HealthChecker" Id=39 TIMED_WAITING on com.tc.util.concurrent.SetOnceFlag@295ac8c2
at java.lang.Object.wait(Native Method)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:247)
at java.lang.Thread.run(Thread.java:745)
"L2_L2:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9530)" Id=38 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7fb33f32> (a sun.nio.ch.Util$2)
- locked <0x5455eaac> (a java.util.Collections$UnmodifiableSet)
- locked <0x7211a5e4> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)" Id=37 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xd726fbc> (a sun.nio.ch.Util$2)
- locked <0x6bde8b8f> (a java.util.Collections$UnmodifiableSet)
- locked <0x10251cd2> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 3_W" Id=36 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x6fe4a5b7> (a sun.nio.ch.Util$2)
- locked <0x71753878> (a java.util.Collections$UnmodifiableSet)
- locked <0x6ee8472a> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 3_R" Id=35 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x561305a9> (a sun.nio.ch.Util$2)
- locked <0x1c10dbc1> (a java.util.Collections$UnmodifiableSet)
- locked <0x447e86c3> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 2_W" Id=34 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x68834871> (a sun.nio.ch.Util$2)
- locked <0x26733685> (a java.util.Collections$UnmodifiableSet)
- locked <0x3ca73f59> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 2_R" Id=33 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xc5ffe90> (a sun.nio.ch.Util$2)
- locked <0x7eedffaa> (a java.util.Collections$UnmodifiableSet)
- locked <0x411529ec> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 1_W" Id=32 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x44cd8b60> (a sun.nio.ch.Util$2)
- locked <0x37954617> (a java.util.Collections$UnmodifiableSet)
- locked <0x2694ffe0> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 1_R" Id=31 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x4e26e897> (a sun.nio.ch.Util$2)
- locked <0x6618ecb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x7b20bb92> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 0_W" Id=30 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7aca30e9> (a sun.nio.ch.Util$2)
- locked <0x3510d77> (a java.util.Collections$UnmodifiableSet)
- locked <0x228d4c89> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 0_R" Id=29 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x5c8f9f61> (a sun.nio.ch.Util$2)
- locked <0x77e9b161> (a java.util.Collections$UnmodifiableSet)
- locked <0x7cd8205e> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"TC Group Manager Handshake timer" Id=28 WAITING on java.util.TaskQueue@3479d248
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"Reconnect timer" Id=27 WAITING on java.util.TaskQueue@14ad2731
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"DSO Lock Object.wait() timer" Id=26 WAITING on java.util.TaskQueue@2e75fe7c
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"TC Memory Monitor" Id=25 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.tc.runtime.TCMemoryManagerImpl$MemoryMonitor.run(TCMemoryManagerImpl.java:140)
"HealthChecker" Id=24 TIMED_WAITING on com.tc.util.concurrent.SetOnceFlag@1392e13c
at java.lang.Object.wait(Native Method)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:247)
at java.lang.Thread.run(Thread.java:745)
"L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)" Id=23 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x69038a94> (a sun.nio.ch.Util$2)
- locked <0x14eaac2a> (a java.util.Collections$UnmodifiableSet)
- locked <0x156d26d5> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)" Id=22 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x11f2ae45> (a sun.nio.ch.Util$2)
- locked <0x6450c1c5> (a java.util.Collections$UnmodifiableSet)
- locked <0x251aba99> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 3_W" Id=21 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x59bd7b61> (a sun.nio.ch.Util$2)
- locked <0x5d397695> (a java.util.Collections$UnmodifiableSet)
- locked <0x3f85ca1e> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 3_R" Id=20 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x4f3b6105> (a sun.nio.ch.Util$2)
- locked <0x7def6df0> (a java.util.Collections$UnmodifiableSet)
- locked <0x6bf43106> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 2_W" Id=19 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7ab0c35c> (a sun.nio.ch.Util$2)
- locked <0x6e97b646> (a java.util.Collections$UnmodifiableSet)
- locked <0x6811d689> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 2_R" Id=18 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2227a68c> (a sun.nio.ch.Util$2)
- locked <0x5e712bb7> (a java.util.Collections$UnmodifiableSet)
- locked <0xd8793b9> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 1_W" Id=17 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x6bc4a91e> (a sun.nio.ch.Util$2)
- locked <0x74c8612c> (a java.util.Collections$UnmodifiableSet)
- locked <0x23b3c091> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 1_R" Id=16 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x12989505> (a sun.nio.ch.Util$2)
- locked <0x5cda5211> (a java.util.Collections$UnmodifiableSet)
- locked <0x6bac99a7> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 0_W" Id=15 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x56267b0b> (a sun.nio.ch.Util$2)
- locked <0x40ec77> (a java.util.Collections$UnmodifiableSet)
- locked <0xe846cb4> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 0_R" Id=14 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x16aad62> (a sun.nio.ch.Util$2)
- locked <0x64716067> (a java.util.Collections$UnmodifiableSet)
- locked <0x63d51bca> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"SampledCounterManager Timer" Id=13 TIMED_WAITING on java.util.TaskQueue@4e192ac8
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"SingleThreadedTimer" Id=12 WAITING on com.tc.services.SingleThreadedTimer@1acc8fd4
at java.lang.Object.wait(Native Method)
at com.tc.services.SingleThreadedTimer.getNextRunnable(SingleThreadedTimer.java:156)
at com.tc.services.SingleThreadedTimer.access$000(SingleThreadedTimer.java:33)
at com.tc.services.SingleThreadedTimer$1.run(SingleThreadedTimer.java:38)
"RMI TCP Accept-0" Id=11 RUNNABLE (in native)
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x7551a6b2> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:389)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:361)
at java.lang.Thread.run(Thread.java:745)
"Attach Listener" Id=5 RUNNABLE
"Signal Dispatcher" Id=4 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3e11f899
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@3959caf9
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
"main" Id=1 WAITING on com.tc.server.TCServerImpl@47954029
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.tc.server.TCServerImpl.waitUntilShutdown(TCServerImpl.java:455)
at com.tc.server.TCServerMain.main(TCServerMain.java:62)
[dump]
***********************************************************************************
com.tc.async.impl.StageManagerImpl
[dump] --> Queue depth: 0 platform_information_request
[dump] --> Queue depth: 0 group_hydrate_message_stage
[dump] --> Queue depth: 0 hydrate_message_stage
[dump] --> Queue depth: 0 passive_replication_ack_stage
[dump] --> Queue depth: 0 group_events_dispatch_stage
[dump] --> Queue depth: 0 request_lock_stage
[dump] --> Queue depth: 0 l2_election_handler_stage
[dump] --> Queue depth: 0 l2_state_change_stage
[dump] --> Queue depth: 0 active_to_passive_driver_stage
[dump] --> Queue depth: 0 server_entity_response_message_stage
[dump] --> Queue depth: 0 receive_group_message_stage
[dump] --> Queue depth: 0 respond_to_request_stage
[dump] --> Queue depth: 0 group_handshake_message_stage
[dump] --> Queue depth: 0 group_discovery_stage
[dump] --> Queue depth: 0 voltron_message_stage
[dump] --> Queue depth: 0 respond_to_lock_request_stage
[dump] --> Queue depth: 0 request_processor_stage
[dump] --> Queue depth: 0 client_handshake_stage
[dump] --> Queue depth: 0 l2_state_message_handler_stage
[dump] --> Queue depth: 0 passive_replication_stage
[dump]
***********************************************************************************
com.tc.objectserver.persistence.Persistor
[dump] com.tc.objectserver.persistence.NullPlatformPersistentStorage@2dd1da1a
[dump]
***********************************************************************************
com.tc.l2.ha.StripeIDStateManagerImpl
[dump] stripeID: StripeID[NULL-ID]
[dump]
***********************************************************************************
com.tc.objectserver.locks.LockManagerImpl
[dump] --> locks: 0
[dump]
***********************************************************************************
--> TCGroupManagerImpl [ Channel to NodeId Map: {}
members: {}
zappedSet: {[] } ]
[dump]
***********************************************************************************
--> L2HACoordinator [ ]
[dump] --> ReplicatedClusterStateMgr--> ReplicatedClusterStateManagerImpl [ ClusterState [ Connections [ [ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]] ] nextAvailChannelID: 1 currentState: State[ ACTIVE-COORDINATOR ] stripeID: StripeID[7a1c76996ff24e19ab669d112b468b96] ] StateManagerImpl:State[ ACTIVE-COORDINATOR ] ]
2017-01-09 23:41:50,436 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.server.TCServerMain - ExitState : CallbackOnExitState[Throwable: class java.lang.AssertionError; RestartNeeded: false]; AutoRestart: true
2017-01-09 23:41:52,439 [CommonShutDownHook] INFO com.terracottatech.tsa - L2 Exiting...
================
client log:
----------------
2017-01-09 23:41:49.279 INFO 7288 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring FrameworkServlet 'dispatcherServlet'
2017-01-09 23:41:49.279 INFO 7288 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started
2017-01-09 23:41:49.306 INFO 7288 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 27 ms
2017-01-09 23:41:49.637 INFO 7288 --- [nio-8080-exec-1] class.com.tc.net.core.TCCommImpl : Comm Worker Threads NOT requested
2017-01-09 23:41:49.652 INFO 7288 --- [nio-8080-exec-1] .ConnectionHealthCheckerImpl: DSO Client : HealthChecker Started
2017-01-09 23:41:49.858 INFO 7288 --- [nio-8080-exec-1] c.c.t.n.p.tcm.CommunicationsManager : HealthCheck CallbackPort Listener started at /0:0:0:0:0:0:0:0:56825
2017-01-09 23:41:50.045 INFO 7288 --- [d727bc5412c5fd5] class.com.tc.net.core.TCConnection : Comms Message Batching enabled
2017-01-09 23:41:50.060 INFO 7288 --- [d727bc5412c5fd5] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(-1.ffffffffffffffffffffffffffffffff.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1014521529: connected: true, closed: false local=127.0.0.1:56826 remote=127.0.0.1:9510 connect=[Mon Jan 09 23:41:50 EST 2017] idle=10ms [0 read, 0 write]
2017-01-09 23:41:50.128 INFO 7288 --- [d727bc5412c5fd5] c.c.t.n.p.t.C. DSO Client : Health monitoring agent started for 127.0.0.1:9510
2017-01-09 23:41:50.141 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.n.p.t.C. DSO Client : HealthCheckCallbackPort verification PASSED for 127.0.0.1:9510(callbackport: 9510)
2017-01-09 23:41:50.141 INFO 7288 --- [d727bc5412c5fd5] c.c.t.p.r.ClientChannelEventController : Got channel event - type: TRANSPORT_CONNECTED_EVENT, event: com.tc.net.protocol.tcm.ChannelEventImpl@180357592[type = TRANSPORT_CONNECTED_EVENT, timestamp = Mon Jan 09 23:41:50 EST 2017, channel = ChannelID[NULL_ID, Status:INIT]:127.0.0.1:56826 <--> 127.0.0.1:9510 remote node : StripeID[NULL-ID]
2017-01-09 23:41:50.142 WARN 7288 --- [d727bc5412c5fd5] c.c.t.n.p.tcm.ClientMessageChannel : Attempt to get the channel ID of an unopened channel - ChannelID=[0]
2017-01-09 23:41:50.142 WARN 7288 --- [d727bc5412c5fd5] c.c.t.n.p.tcm.ClientMessageChannel : Attempt to get the channel ID of an unopened channel - ChannelID=[0]
2017-01-09 23:41:50.143 WARN 7288 --- [d727bc5412c5fd5] c.c.t.n.p.tcm.ClientMessageChannel : Attempt to get the channel ID of an unopened channel - ChannelID=[0]
2017-01-09 23:41:50.143 INFO 7288 --- [d727bc5412c5fd5] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Connected: Unpausing from PAUSED
2017-01-09 23:41:50.150 INFO 7288 --- [d727bc5412c5fd5] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Sending handshake message
2017-01-09 23:41:50.151 INFO 7288 --- [d727bc5412c5fd5] c.c.t.p.r.ClientChannelEventController : Got channel event - type: CHANNEL_OPENED_EVENT, event: com.tc.net.protocol.tcm.ChannelEventImpl@727073050[type = CHANNEL_OPENED_EVENT, timestamp = Mon Jan 09 23:41:50 EST 2017, channel = ChannelID=[0]:127.0.0.1:56826 <--> 127.0.0.1:9510 remote node : StripeID[NULL-ID]
2017-01-09 23:41:50.159 INFO 7288 --- [ation_stage, 0)] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Received Handshake ack
2017-01-09 23:41:50.162 INFO 7288 --- [ation_stage, 0)] class.com.tc.cluster.ClusterImpl : NODE_JOINED ClientID[0]
2017-01-09 23:41:50.164 INFO 7288 --- [d727bc5412c5fd5] console.logger : Connection successfully established to server at 127.0.0.1:9510
2017-01-09 23:41:50.164 INFO 7288 --- [d727bc5412c5fd5] customer.logger.tsa : Connection successfully established to server at 127.0.0.1:9510
2017-01-09 23:41:52.778 INFO 7288 --- [:0:0:0:0:56825)] class.com.tc.net.core.TCConnection : error reading from channel java.nio.channels.SocketChannel[connected local=/127.0.0.1:56826 remote=localhost/127.0.0.1:9510]: An existing connection was forcibly closed by the remote host
2017-01-09 23:41:52.779 INFO 7288 --- [:0:0:0:0:56825)] c.com.tc.net.core.TCConnectionManager : error event on connection com.tc.net.core.TCConnectionImpl@1014521529: connected: true, closed: false local=127.0.0.1:56826 remote=127.0.0.1:9510 connect=[Mon Jan 09 23:41:50 EST 2017] idle=2461ms [713 read, 1330 write]: An existing connection was forcibly closed by the remote host
2017-01-09 23:41:52.780 WARN 7288 --- [:0:0:0:0:56825)] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: CLOSE EVENT : com.tc.net.core.TCConnectionImpl@1014521529: connected: false, closed: true local=127.0.0.1:56826 remote=127.0.0.1:9510 connect=[Mon Jan 09 23:41:50 EST 2017] idle=2462ms [713 read, 1330 write]. STATUS : ESTABLISHED
2017-01-09 23:41:52.782 INFO 7288 --- [:0:0:0:0:56825)] .ConnectionHealthCheckerImpl: DSO Client : Connection to [127.0.0.1:9510] DISCONNECTED. Health Monitoring for this node is now disabled.
2017-01-09 23:41:52.788 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.n.p.tcm.ClientMessageChannel : ClientMessageChannel moves to SessionID=[1] for remote node StripeID[NULL-ID]
2017-01-09 23:41:52.813 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.p.r.ClientChannelEventController : Got channel event - type: TRANSPORT_DISCONNECTED_EVENT, event: com.tc.net.protocol.tcm.ChannelEventImpl@1938512144[type = TRANSPORT_DISCONNECTED_EVENT, timestamp = Mon Jan 09 23:41:52 EST 2017, channel = ChannelID=[0]:127.0.0.1:56826 <--> 127.0.0.1:9510 remote node : StripeID[NULL-ID]
2017-01-09 23:41:52.813 INFO 7288 --- [9510 -462585769] ientConnectionEstablisher$AsyncReconnect : Connection establisher starting. 462585769
2017-01-09 23:41:52.813 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Disconnected: Pausing from RUNNING. Disconnect count: false
2017-01-09 23:41:52.814 INFO 7288 --- [9510 -462585769] ientConnectionEstablisher$AsyncReconnect : Handling connection request: ConnectionRequest [type=RECONNECT, cmt=com.tc.net.protocol.transport.ClientMessageTransport@489f5ffa]
2017-01-09 23:41:52.814 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: ClientHandshakeManager moves to SessionID=[1]
2017-01-09 23:41:52.820 WARN 7288 --- [9510 -462585769] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Reconnect attempt 0 of unlimited reconnect tries to localhost:9510, timeout=10000
2017-01-09 23:41:53.823 WARN 7288 --- [9510 -462585769] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Connection refused: no further information [lossy interval: 10000ms]
2017-01-09 23:42:03.838 WARN 7288 --- [9510 -462585769] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Connection refused: no further information [lossy interval: 10000ms]
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tc-config xmlns="http://www.terracotta.org/config">
<servers secure="false">
<server host="localhost" name="testing" bind="0.0.0.0">
<logs>C:\Users\terracotta\server-logs</logs>
<tsa-port bind="0.0.0.0">9510</tsa-port>
<management-port bind="0.0.0.0">9540</management-port>
<tsa-group-port bind="0.0.0.0">9530</tsa-group-port>
</server>
<client-reconnect-window>120</client-reconnect-window>
</servers>
</tc-config>
=======================
server logs
------------------------------
2017-01-09 23:40:29,855 [main] INFO com.tc.server.TCServer - Server started as testing
2017-01-09 23:40:34,861 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.ElectionManagerImpl - Election Complete : [Enrollment [ NodeID[localhost:9510], isNew = true, weights = 0,0,116,-6093981732416992524 ]] : State[ Election-Complete ]
2017-01-09 23:40:34,861 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.StateManagerImpl - Becoming State[ ACTIVE-COORDINATOR ]
2017-01-09 23:40:34,861 [WorkerThread(l2_election_handler_stage, 0)] INFO com.terracottatech.console - Becoming State[ ACTIVE-COORDINATOR ]
2017-01-09 23:40:34,861 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - putToStore StripeID[7a1c76996ff24e19ab669d112b468b96]
2017-01-09 23:40:34,861 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.l2.ha.StripeIDStateManagerImpl - Notify local StripeID ready
2017-01-09 23:40:34,876 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.ElectionManagerImpl - Declared as Winner: Winner is : Enrollment [ NodeID[localhost:9510], isNew = true, weights = 0,0,116,-6093981732416992524 ]
2017-01-09 23:40:34,876 [WorkerThread(l2_state_change_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Starting TSA services...
2017-01-09 23:40:34,892 [WorkerThread(l2_state_change_stage, 0)] INFO com.terracottatech.console - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.
2017-01-09 23:40:35,292 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCConnection - Comms Message Batching enabled
2017-01-09 23:40:35,346 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@737495433: connected: true, closed: false local=127.0.0.1:9510 remote=127.0.0.1:56803 connect=[Mon Jan 09 23:40:35 EST 2017] idle=32ms [0 read, 0 write]
2017-01-09 23:40:35,346 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - "Client Cannot Reconnect. Connection attempts from the Terracotta node at 127.0.0.1:56803 are being rejected by the Terracotta server array. Reason: Stack for ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[] not found."
2017-01-09 23:40:35,361 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[]: CLOSE EVENT : com.tc.net.core.TCConnectionImpl@737495433: connected: false, closed: true local=127.0.0.1:9510 remote=127.0.0.1:56803 connect=[Mon Jan 09 23:40:35 EST 2017] idle=0ms [152 read, 495 write]. STATUS : START
2017-01-09 23:40:35,361 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.ec668f0f9e384f0485ab8434457c9d6d.ecf1e7e5-807e-4a08-92e6-b0313259b691-159869d83fc.User)[]: closing down connection - com.tc.net.core.TCConnectionImpl@737495433: connected: false, closed: true local=127.0.0.1:9510 remote=127.0.0.1:56803 connect=[Mon Jan 09 23:40:35 EST 2017] idle=0ms [152 read, 495 write]
2017-01-09 23:41:50,092 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1651296138: connected: true, closed: false local=127.0.0.1:9510 remote=127.0.0.1:56826 connect=[Mon Jan 09 23:41:50 EST 2017] idle=2ms [0 read, 0 write]
2017-01-09 23:41:50,113 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - Allocated connection license for jvm a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a; ConnectionPolicy[maxConnections=2147483647, connectedJvmCount=1]
2017-01-09 23:41:50,114 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - New connection [0] from jvm a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a
2017-01-09 23:41:50,114 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ServerStackProvider - User null successfully authenticated
2017-01-09 23:41:50,115 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.TCWorkerCommManager - Selecting [L2_L1:TCWorkerComm # 0, FD, wt:0] from [[L2_L1:TCWorkerComm # 0, FD, wt:0], [L2_L1:TCWorkerComm # 1, FD, wt:0], [L2_L1:TCWorkerComm # 2, FD, wt:0], [L2_L1:TCWorkerComm # 3, FD, wt:0]]
2017-01-09 23:41:50,116 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.core.CoreNIOServices - selection key cancelled key@1280936924
2017-01-09 23:41:50,128 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TSA Server - Health monitoring agent started for 127.0.0.1:56826
2017-01-09 23:41:50,135 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TSA Server - HealthCheckCallbackPort verification PASSED for 127.0.0.1:56826(callbackport: 56825)
2017-01-09 23:41:50,154 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Handling client handshake for ClientID[0]
2017-01-09 23:41:50,154 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Sending handshake acknowledgement to ClientID[0]
2017-01-09 23:41:50,246 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.objectserver.entity.ManagedEntityImpl - Client:ClientID[0] Invoking lifecycle CREATE_ENTITY on EntityID{className='org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient', entityName='VoltronReadWriteLock-EhcacheClientEntityFactory-AccessLock-testing'}
2017-01-09 23:41:50,269 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.objectserver.entity.ManagedEntityImpl - Client:ClientID[0] Invoking lifecycle FETCH_ENTITY on EntityID{className='org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient', entityName='VoltronReadWriteLock-EhcacheClientEntityFactory-AccessLock-testing'}
2017-01-09 23:41:50,386 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.objectserver.entity.ManagedEntityImpl - Client:ClientID[0] Invoking lifecycle CREATE_ENTITY on EntityID{className='org.ehcache.clustered.client.internal.EhcacheClientEntity', entityName='testing'}
2017-01-09 23:41:50,413 [WorkerThread(request_processor_stage, 2, 4)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(request_processor_stage, 2, 4),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
java.lang.AssertionError: Server failed to retrieve EhcacheStateService.
at org.ehcache.clustered.server.EhcacheActiveEntity.<init>(EhcacheActiveEntity.java:181)
at org.ehcache.clustered.server.EhcacheServerEntityService.createActiveEntity(EhcacheServerEntityService.java:50)
at org.ehcache.clustered.server.EhcacheServerEntityService.createActiveEntity(EhcacheServerEntityService.java:32)
at com.tc.objectserver.entity.ManagedEntityImpl.createEntity(ManagedEntityImpl.java:601)
at com.tc.objectserver.entity.ManagedEntityImpl.invokeLifecycleOperation(ManagedEntityImpl.java:365)
at com.tc.objectserver.entity.ManagedEntityImpl.lambda$processLifecycleEntity$61(ManagedEntityImpl.java:224)
at com.tc.objectserver.entity.ManagedEntityImpl$$Lambda$26/1625390904.run(Unknown Source)
at com.tc.objectserver.entity.ManagedEntityImpl$SchedulingRunnable.run(ManagedEntityImpl.java:958)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.invoke(RequestProcessor.java:161)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.run(RequestProcessor.java:154)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:27)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:23)
at com.tc.async.impl.StageQueueImpl$FlushingHandledContext.runWithHandler(StageQueueImpl.java:527)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:197)
2017-01-09 23:41:50,428 [WorkerThread(request_processor_stage, 2, 4)] ERROR com.tc.logging.ThreadDumpHandler - Mon Jan 09 23:41:50 EST 2017
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):
"Dump On Timeout Timer" Id=65 TIMED_WAITING on java.util.TaskQueue@5f9b9a75
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"WorkerThread(client_handshake_stage, 0)" Id=64 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a250ab7
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(passive_replication_ack_stage, 0)" Id=63 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a9b6f96
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(respond_to_request_stage, 0)" Id=62 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30c2e550
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(voltron_message_stage, 0)" Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3309890f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(active_to_passive_driver_stage, 0)" Id=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11db4d6b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"Static Member discovery" Id=58 WAITING on com.tc.net.groups.TCGroupMemberDiscoveryStatic@301d6719
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.tc.net.groups.TCGroupMemberDiscoveryStatic.pauseDiscovery(TCGroupMemberDiscoveryStatic.java:277)
at com.tc.net.groups.TCGroupMemberDiscoveryStatic$1.run(TCGroupMemberDiscoveryStatic.java:182)
at java.lang.Thread.run(Thread.java:745)
"WorkerThread(l2_state_message_handler_stage, 0)" Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1752fc2f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 3, 4)" Id=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@686c4e3b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 2, 4)" Id=55 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
at com.tc.util.runtime.ThreadDumpUtil.getThreadDump(ThreadDumpUtil.java:143)
at com.tc.logging.ThreadDumpHandler.callbackOnExit(ThreadDumpHandler.java:41)
at com.tc.lang.ThrowableHandlerImpl.handleDefaultException(ThrowableHandlerImpl.java:211)
- locked <0x2397e648> (a java.lang.Object)
at com.tc.lang.ThrowableHandlerImpl.handleThrowable(ThrowableHandlerImpl.java:147)
at com.tc.lang.TCThreadGroup.uncaughtException(TCThreadGroup.java:38)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1952)
"WorkerThread(request_processor_stage, 1, 4)" Id=54 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78e9688c
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 0, 4)" Id=53 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78da503a
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_discovery_stage, 0)" Id=52 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72cdc48c
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_handshake_message_stage, 0)" Id=51 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ef4d93b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(receive_group_message_stage, 0)" Id=50 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d6a7161
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(server_entity_response_message_stage, 0)" Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@771ba176
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(l2_state_change_stage, 0)" Id=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b6ee2bd
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(l2_election_handler_stage, 0)" Id=47 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@239bc3ea
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_events_dispatch_stage, 0)" Id=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d745f81
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 3, 4)" Id=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ce5b48
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 2, 4)" Id=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b293a06
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 1, 4)" Id=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66502e1a
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 0, 4)" Id=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6957ed06
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_hydrate_message_stage, 0)" Id=41 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1cb6ef7f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(platform_information_request, 0)" Id=40 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4e1d0fe5
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.StageQueueImpl$SourceQueueImpl.poll(StageQueueImpl.java:353)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"HealthChecker" Id=39 TIMED_WAITING on com.tc.util.concurrent.SetOnceFlag@295ac8c2
at java.lang.Object.wait(Native Method)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:247)
at java.lang.Thread.run(Thread.java:745)
"L2_L2:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9530)" Id=38 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7fb33f32> (a sun.nio.ch.Util$2)
- locked <0x5455eaac> (a java.util.Collections$UnmodifiableSet)
- locked <0x7211a5e4> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9530)" Id=37 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xd726fbc> (a sun.nio.ch.Util$2)
- locked <0x6bde8b8f> (a java.util.Collections$UnmodifiableSet)
- locked <0x10251cd2> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 3_W" Id=36 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x6fe4a5b7> (a sun.nio.ch.Util$2)
- locked <0x71753878> (a java.util.Collections$UnmodifiableSet)
- locked <0x6ee8472a> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 3_R" Id=35 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x561305a9> (a sun.nio.ch.Util$2)
- locked <0x1c10dbc1> (a java.util.Collections$UnmodifiableSet)
- locked <0x447e86c3> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 2_W" Id=34 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x68834871> (a sun.nio.ch.Util$2)
- locked <0x26733685> (a java.util.Collections$UnmodifiableSet)
- locked <0x3ca73f59> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 2_R" Id=33 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xc5ffe90> (a sun.nio.ch.Util$2)
- locked <0x7eedffaa> (a java.util.Collections$UnmodifiableSet)
- locked <0x411529ec> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 1_W" Id=32 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x44cd8b60> (a sun.nio.ch.Util$2)
- locked <0x37954617> (a java.util.Collections$UnmodifiableSet)
- locked <0x2694ffe0> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 1_R" Id=31 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x4e26e897> (a sun.nio.ch.Util$2)
- locked <0x6618ecb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x7b20bb92> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 0_W" Id=30 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7aca30e9> (a sun.nio.ch.Util$2)
- locked <0x3510d77> (a java.util.Collections$UnmodifiableSet)
- locked <0x228d4c89> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L2:TCWorkerComm # 0_R" Id=29 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x5c8f9f61> (a sun.nio.ch.Util$2)
- locked <0x77e9b161> (a java.util.Collections$UnmodifiableSet)
- locked <0x7cd8205e> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"TC Group Manager Handshake timer" Id=28 WAITING on java.util.TaskQueue@3479d248
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"Reconnect timer" Id=27 WAITING on java.util.TaskQueue@14ad2731
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"DSO Lock Object.wait() timer" Id=26 WAITING on java.util.TaskQueue@2e75fe7c
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"TC Memory Monitor" Id=25 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.tc.runtime.TCMemoryManagerImpl$MemoryMonitor.run(TCMemoryManagerImpl.java:140)
"HealthChecker" Id=24 TIMED_WAITING on com.tc.util.concurrent.SetOnceFlag@1392e13c
at java.lang.Object.wait(Native Method)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:247)
at java.lang.Thread.run(Thread.java:745)
"L2_L1:TCComm Main Selector Thread_W (listen 0:0:0:0:0:0:0:0:9510)" Id=23 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x69038a94> (a sun.nio.ch.Util$2)
- locked <0x14eaac2a> (a java.util.Collections$UnmodifiableSet)
- locked <0x156d26d5> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9510)" Id=22 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x11f2ae45> (a sun.nio.ch.Util$2)
- locked <0x6450c1c5> (a java.util.Collections$UnmodifiableSet)
- locked <0x251aba99> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 3_W" Id=21 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x59bd7b61> (a sun.nio.ch.Util$2)
- locked <0x5d397695> (a java.util.Collections$UnmodifiableSet)
- locked <0x3f85ca1e> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 3_R" Id=20 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x4f3b6105> (a sun.nio.ch.Util$2)
- locked <0x7def6df0> (a java.util.Collections$UnmodifiableSet)
- locked <0x6bf43106> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 2_W" Id=19 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7ab0c35c> (a sun.nio.ch.Util$2)
- locked <0x6e97b646> (a java.util.Collections$UnmodifiableSet)
- locked <0x6811d689> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 2_R" Id=18 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2227a68c> (a sun.nio.ch.Util$2)
- locked <0x5e712bb7> (a java.util.Collections$UnmodifiableSet)
- locked <0xd8793b9> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 1_W" Id=17 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x6bc4a91e> (a sun.nio.ch.Util$2)
- locked <0x74c8612c> (a java.util.Collections$UnmodifiableSet)
- locked <0x23b3c091> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 1_R" Id=16 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x12989505> (a sun.nio.ch.Util$2)
- locked <0x5cda5211> (a java.util.Collections$UnmodifiableSet)
- locked <0x6bac99a7> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 0_W" Id=15 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x56267b0b> (a sun.nio.ch.Util$2)
- locked <0x40ec77> (a java.util.Collections$UnmodifiableSet)
- locked <0xe846cb4> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"L2_L1:TCWorkerComm # 0_R" Id=14 RUNNABLE (in native)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x16aad62> (a sun.nio.ch.Util$2)
- locked <0x64716067> (a java.util.Collections$UnmodifiableSet)
- locked <0x63d51bca> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:566)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:310)
"SampledCounterManager Timer" Id=13 TIMED_WAITING on java.util.TaskQueue@4e192ac8
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"SingleThreadedTimer" Id=12 WAITING on com.tc.services.SingleThreadedTimer@1acc8fd4
at java.lang.Object.wait(Native Method)
at com.tc.services.SingleThreadedTimer.getNextRunnable(SingleThreadedTimer.java:156)
at com.tc.services.SingleThreadedTimer.access$000(SingleThreadedTimer.java:33)
at com.tc.services.SingleThreadedTimer$1.run(SingleThreadedTimer.java:38)
"RMI TCP Accept-0" Id=11 RUNNABLE (in native)
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x7551a6b2> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:389)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:361)
at java.lang.Thread.run(Thread.java:745)
"Attach Listener" Id=5 RUNNABLE
"Signal Dispatcher" Id=4 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3e11f899
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@3959caf9
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
"main" Id=1 WAITING on com.tc.server.TCServerImpl@47954029
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.tc.server.TCServerImpl.waitUntilShutdown(TCServerImpl.java:455)
at com.tc.server.TCServerMain.main(TCServerMain.java:62)
[dump]
***********************************************************************************
com.tc.async.impl.StageManagerImpl
[dump] --> Queue depth: 0 platform_information_request
[dump] --> Queue depth: 0 group_hydrate_message_stage
[dump] --> Queue depth: 0 hydrate_message_stage
[dump] --> Queue depth: 0 passive_replication_ack_stage
[dump] --> Queue depth: 0 group_events_dispatch_stage
[dump] --> Queue depth: 0 request_lock_stage
[dump] --> Queue depth: 0 l2_election_handler_stage
[dump] --> Queue depth: 0 l2_state_change_stage
[dump] --> Queue depth: 0 active_to_passive_driver_stage
[dump] --> Queue depth: 0 server_entity_response_message_stage
[dump] --> Queue depth: 0 receive_group_message_stage
[dump] --> Queue depth: 0 respond_to_request_stage
[dump] --> Queue depth: 0 group_handshake_message_stage
[dump] --> Queue depth: 0 group_discovery_stage
[dump] --> Queue depth: 0 voltron_message_stage
[dump] --> Queue depth: 0 respond_to_lock_request_stage
[dump] --> Queue depth: 0 request_processor_stage
[dump] --> Queue depth: 0 client_handshake_stage
[dump] --> Queue depth: 0 l2_state_message_handler_stage
[dump] --> Queue depth: 0 passive_replication_stage
[dump]
***********************************************************************************
com.tc.objectserver.persistence.Persistor
[dump] com.tc.objectserver.persistence.NullPlatformPersistentStorage@2dd1da1a
[dump]
***********************************************************************************
com.tc.l2.ha.StripeIDStateManagerImpl
[dump] stripeID: StripeID[NULL-ID]
[dump]
***********************************************************************************
com.tc.objectserver.locks.LockManagerImpl
[dump] --> locks: 0
[dump]
***********************************************************************************
--> TCGroupManagerImpl [ Channel to NodeId Map: {}
members: {}
zappedSet: {[] } ]
[dump]
***********************************************************************************
--> L2HACoordinator [ ]
[dump] --> ReplicatedClusterStateMgr--> ReplicatedClusterStateManagerImpl [ ClusterState [ Connections [ [ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]] ] nextAvailChannelID: 1 currentState: State[ ACTIVE-COORDINATOR ] stripeID: StripeID[7a1c76996ff24e19ab669d112b468b96] ] StateManagerImpl:State[ ACTIVE-COORDINATOR ] ]
2017-01-09 23:41:50,436 [WorkerThread(request_processor_stage, 2, 4)] INFO com.tc.server.TCServerMain - ExitState : CallbackOnExitState[Throwable: class java.lang.AssertionError; RestartNeeded: false]; AutoRestart: true
2017-01-09 23:41:52,439 [CommonShutDownHook] INFO com.terracottatech.tsa - L2 Exiting...
================
client log:
----------------
2017-01-09 23:41:49.279 INFO 7288 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring FrameworkServlet 'dispatcherServlet'
2017-01-09 23:41:49.279 INFO 7288 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started
2017-01-09 23:41:49.306 INFO 7288 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 27 ms
2017-01-09 23:41:49.637 INFO 7288 --- [nio-8080-exec-1] class.com.tc.net.core.TCCommImpl : Comm Worker Threads NOT requested
2017-01-09 23:41:49.652 INFO 7288 --- [nio-8080-exec-1] .ConnectionHealthCheckerImpl: DSO Client : HealthChecker Started
2017-01-09 23:41:49.858 INFO 7288 --- [nio-8080-exec-1] c.c.t.n.p.tcm.CommunicationsManager : HealthCheck CallbackPort Listener started at /0:0:0:0:0:0:0:0:56825
2017-01-09 23:41:50.045 INFO 7288 --- [d727bc5412c5fd5] class.com.tc.net.core.TCConnection : Comms Message Batching enabled
2017-01-09 23:41:50.060 INFO 7288 --- [d727bc5412c5fd5] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(-1.ffffffffffffffffffffffffffffffff.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Attaching new connection: com.tc.net.core.TCConnectionImpl@1014521529: connected: true, closed: false local=127.0.0.1:56826 remote=127.0.0.1:9510 connect=[Mon Jan 09 23:41:50 EST 2017] idle=10ms [0 read, 0 write]
2017-01-09 23:41:50.128 INFO 7288 --- [d727bc5412c5fd5] c.c.t.n.p.t.C. DSO Client : Health monitoring agent started for 127.0.0.1:9510
2017-01-09 23:41:50.141 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.n.p.t.C. DSO Client : HealthCheckCallbackPort verification PASSED for 127.0.0.1:9510(callbackport: 9510)
2017-01-09 23:41:50.141 INFO 7288 --- [d727bc5412c5fd5] c.c.t.p.r.ClientChannelEventController : Got channel event - type: TRANSPORT_CONNECTED_EVENT, event: com.tc.net.protocol.tcm.ChannelEventImpl@180357592[type = TRANSPORT_CONNECTED_EVENT, timestamp = Mon Jan 09 23:41:50 EST 2017, channel = ChannelID[NULL_ID, Status:INIT]:127.0.0.1:56826 <--> 127.0.0.1:9510 remote node : StripeID[NULL-ID]
2017-01-09 23:41:50.142 WARN 7288 --- [d727bc5412c5fd5] c.c.t.n.p.tcm.ClientMessageChannel : Attempt to get the channel ID of an unopened channel - ChannelID=[0]
2017-01-09 23:41:50.142 WARN 7288 --- [d727bc5412c5fd5] c.c.t.n.p.tcm.ClientMessageChannel : Attempt to get the channel ID of an unopened channel - ChannelID=[0]
2017-01-09 23:41:50.143 WARN 7288 --- [d727bc5412c5fd5] c.c.t.n.p.tcm.ClientMessageChannel : Attempt to get the channel ID of an unopened channel - ChannelID=[0]
2017-01-09 23:41:50.143 INFO 7288 --- [d727bc5412c5fd5] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Connected: Unpausing from PAUSED
2017-01-09 23:41:50.150 INFO 7288 --- [d727bc5412c5fd5] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Sending handshake message
2017-01-09 23:41:50.151 INFO 7288 --- [d727bc5412c5fd5] c.c.t.p.r.ClientChannelEventController : Got channel event - type: CHANNEL_OPENED_EVENT, event: com.tc.net.protocol.tcm.ChannelEventImpl@727073050[type = CHANNEL_OPENED_EVENT, timestamp = Mon Jan 09 23:41:50 EST 2017, channel = ChannelID=[0]:127.0.0.1:56826 <--> 127.0.0.1:9510 remote node : StripeID[NULL-ID]
2017-01-09 23:41:50.159 INFO 7288 --- [ation_stage, 0)] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Received Handshake ack
2017-01-09 23:41:50.162 INFO 7288 --- [ation_stage, 0)] class.com.tc.cluster.ClusterImpl : NODE_JOINED ClientID[0]
2017-01-09 23:41:50.164 INFO 7288 --- [d727bc5412c5fd5] console.logger : Connection successfully established to server at 127.0.0.1:9510
2017-01-09 23:41:50.164 INFO 7288 --- [d727bc5412c5fd5] customer.logger.tsa : Connection successfully established to server at 127.0.0.1:9510
2017-01-09 23:41:52.778 INFO 7288 --- [:0:0:0:0:56825)] class.com.tc.net.core.TCConnection : error reading from channel java.nio.channels.SocketChannel[connected local=/127.0.0.1:56826 remote=localhost/127.0.0.1:9510]: An existing connection was forcibly closed by the remote host
2017-01-09 23:41:52.779 INFO 7288 --- [:0:0:0:0:56825)] c.com.tc.net.core.TCConnectionManager : error event on connection com.tc.net.core.TCConnectionImpl@1014521529: connected: true, closed: false local=127.0.0.1:56826 remote=127.0.0.1:9510 connect=[Mon Jan 09 23:41:50 EST 2017] idle=2461ms [713 read, 1330 write]: An existing connection was forcibly closed by the remote host
2017-01-09 23:41:52.780 WARN 7288 --- [:0:0:0:0:56825)] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: CLOSE EVENT : com.tc.net.core.TCConnectionImpl@1014521529: connected: false, closed: true local=127.0.0.1:56826 remote=127.0.0.1:9510 connect=[Mon Jan 09 23:41:50 EST 2017] idle=2462ms [713 read, 1330 write]. STATUS : ESTABLISHED
2017-01-09 23:41:52.782 INFO 7288 --- [:0:0:0:0:56825)] .ConnectionHealthCheckerImpl: DSO Client : Connection to [127.0.0.1:9510] DISCONNECTED. Health Monitoring for this node is now disabled.
2017-01-09 23:41:52.788 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.n.p.tcm.ClientMessageChannel : ClientMessageChannel moves to SessionID=[1] for remote node StripeID[NULL-ID]
2017-01-09 23:41:52.813 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.p.r.ClientChannelEventController : Got channel event - type: TRANSPORT_DISCONNECTED_EVENT, event: com.tc.net.protocol.tcm.ChannelEventImpl@1938512144[type = TRANSPORT_DISCONNECTED_EVENT, timestamp = Mon Jan 09 23:41:52 EST 2017, channel = ChannelID=[0]:127.0.0.1:56826 <--> 127.0.0.1:9510 remote node : StripeID[NULL-ID]
2017-01-09 23:41:52.813 INFO 7288 --- [9510 -462585769] ientConnectionEstablisher$AsyncReconnect : Connection establisher starting. 462585769
2017-01-09 23:41:52.813 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: Disconnected: Pausing from RUNNING. Disconnect count: false
2017-01-09 23:41:52.814 INFO 7288 --- [9510 -462585769] ientConnectionEstablisher$AsyncReconnect : Handling connection request: ConnectionRequest [type=RECONNECT, cmt=com.tc.net.protocol.transport.ClientMessageTransport@489f5ffa]
2017-01-09 23:41:52.814 INFO 7288 --- [:0:0:0:0:56825)] c.c.t.o.h.ClientHandshakeManagerImpl : ClientID[0]: ClientHandshakeManager moves to SessionID=[1]
2017-01-09 23:41:52.820 WARN 7288 --- [9510 -462585769] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Reconnect attempt 0 of unlimited reconnect tries to localhost:9510, timeout=10000
2017-01-09 23:41:53.823 WARN 7288 --- [9510 -462585769] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Connection refused: no further information [lossy interval: 10000ms]
2017-01-09 23:42:03.838 WARN 7288 --- [9510 -462585769] c.c.t.n.p.t.ClientMessageTransport : ConnectionID(0.7a1c76996ff24e19ab669d112b468b96.a2503ef9-d30c-41c0-97d3-d6f9c57acc74-15986ae428a.User)[]: Connection refused: no further information [lossy interval: 10000ms]
James House
Feb 4, 2017, 12:14:12 PM2/4/17
to terracotta-oss
Nadeem Amin
Jan 3, 2018, 7:48:52 PM1/3/18
to terracotta-oss
i am getting a similar error. was there a resolution to the issue mentioned in this thread ?
2018-01-03 19:39:13,068 [main] INFO com.tc.server.TCServerMain - Terracotta 5.3.2, as of 2017-08-25 at 14:11:55 PDT (Revision 221ca10a1ccc60b459d50f4d332d86930752b79c from UNKNOWN)
2018-01-03 19:39:13,074 [main] INFO com.tc.server.TCServerMain - PID is 25298
2018-01-03 19:39:13,079 [main] INFO com.tc.properties.TCPropertiesImpl - Loading default properties from tc.properties
2018-01-03 19:39:13,084 [main] INFO com.tc.properties.TCPropertiesImpl - The property "l2.dump.on.exception.timeout" was read before initialization completed. "l2.dump.on.exception.timeout" = 30
2018-01-03 19:39:13,097 [main] INFO com.tc.classloader.ServiceLocator - Entity/Service apis will be loaded from /data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/api
2018-01-03 19:39:13,097 [main] INFO com.tc.classloader.ServiceLocator - Entity/Service implementations will be loaded from /data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib
2018-01-03 19:39:13,194 [main] INFO com.tc.properties.TCPropertiesImpl - The property "tc.config.total.timeout" was read before initialization completed. "tc.config.total.timeout" = 300000
2018-01-03 19:39:13,195 [main] INFO com.tc.properties.TCPropertiesImpl - The property "tc.config.getFromSource.timeout" was read before initialization completed. "tc.config.getFromSource.timeout" = 30000
2018-01-03 19:39:13,202 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - Attempting to load configuration from the file at '/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/conf/tc-config.xml'...
2018-01-03 19:39:13,575 [main] INFO org.terracotta.console - Successfully loaded base configuration from file at '/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/conf/tc-config.xml'.
2018-01-03 19:39:13,612 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - The configuration specified by 'base configuration from file at '/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/conf/tc-config.xml'':
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tc-config xmlns="http://www.terracotta.org/config">
<plugins>
<config>
<ohr:offheap-resources xmlns:ohr="http://www.terracotta.org/config/offheap-resource">
<ohr:resource name="primary-server-resource" unit="MB">10240</ohr:resource>
<ohr:resource name="main" unit="MB">12288</ohr:resource>
</ohr:offheap-resources>
</config>
</plugins>
<servers>
<server host="mainhost" name="sat-node-b" bind="0.0.0.0">
<logs>/usr/local/home/bvs/terracotta-logs</logs>
<tsa-port bind="mainhost">8090</tsa-port>
<tsa-group-port bind="0.0.0.0">9430</tsa-group-port>
</server>
<server host="alternatehost" name="sat-node-a" bind="0.0.0.0">
<logs>/usr/local/home/bvs/sat-node-a-logs</logs>
<tsa-port bind="0.0.0.0">8090</tsa-port>
<tsa-group-port bind="0.0.0.0">9430</tsa-group-port>
</server>
<client-reconnect-window>30</client-reconnect-window>
</servers>
</tc-config>
2018-01-03 19:39:13,622 [main] INFO com.tc.l2.logging.TCLogbackLogging - Log file: /usr/local/home/bvs/terracotta-logs/terracotta.server.log
2018-01-03 19:39:13,647 [main] INFO com.tc.l2.logging.TCLogbackLogging - All Java System Properties for this Terracotta instance:
========================================================================
JVM arguments: [-XX:MaxDirectMemorySize=9223372036854775807, -Xms10g, -Xmx15g, -XX:+HeapDumpOnOutOfMemoryError, -Dtc.install-root=/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server]
awt.toolkit : sun.awt.X11.XToolkit
file.encoding : UTF-8
file.encoding.pkg : sun.io
file.separator : /
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.printerjob : sun.print.PSPrinterJob
java.class.path : /data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/lib/tc.jar::/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/client-message-tracker-5.3.1.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/ehcache-common-3.4.0.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/ehcache-server-3.4.0.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/mnm-common-5.3.1.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/mnm-server-5.3.1.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/monitoring-service-api-5.3.1.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/offheap-resource-5.3.1.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/offheap-store-2.3.2.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/runnel-5.3.1.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/plugins/lib/slf4j-api-1.7.7.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/lib/slf4j-api-1.7.7.jar:/data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/lib
java.class.version : 52.0
java.endorsed.dirs : /usr/local/home/bvs/jdk1.8.0_131/jre/lib/endorsed
java.ext.dirs : /usr/local/home/bvs/jdk1.8.0_131/jre/lib/ext:/usr/java/packages/lib/ext
java.home : /usr/local/home/bvs/jdk1.8.0_131/jre
java.io.tmpdir : /tmp
java.library.path : /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.runtime.name : Java(TM) SE Runtime Environment
java.runtime.version : 1.8.0_131-b11
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 1.8
java.vendor : Oracle Corporation
java.vendor.url : http://java.oracle.com/
java.vendor.url.bug : http://bugreport.sun.com/bugreport/
java.version : 1.8.0_131
java.vm.info : mixed mode
java.vm.name : Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 1.8
java.vm.vendor : Oracle Corporation
java.vm.version : 25.131-b11
line.separator :
os.arch : amd64
os.name : Linux
os.version : 3.10.0-693.11.1.el7.x86_64
path.separator : :
sun.arch.data.model : 64
sun.boot.class.path : /usr/local/home/bvs/jdk1.8.0_131/jre/lib/resources.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/lib/rt.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/lib/sunrsasign.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/lib/jsse.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/lib/jce.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/lib/charsets.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/lib/jfr.jar:/usr/local/home/bvs/jdk1.8.0_131/jre/classes
sun.boot.library.path : /usr/local/home/bvs/jdk1.8.0_131/jre/lib/amd64
sun.cpu.endian : little
sun.cpu.isalist :
sun.io.unicode.encoding : UnicodeLittle
sun.java.command : com.tc.server.TCServerMain
sun.java.launcher : SUN_STANDARD
sun.jnu.encoding : UTF-8
sun.management.compiler : HotSpot 64-Bit Tiered Compilers
sun.os.patch.level : unknown
tc.install-root : /data/biometrics-exit/ehcache-clustered-3.4.0-kit/server
user.country : US
user.dir : /data/biometrics-exit/ehcache-clustered-3.4.0-kit/server/bin
user.home : /usr/local/home/bvs
user.language : en
user.name : bvs
user.timezone : America/New_York
========================================================================
2018-01-03 19:39:13,671 [main] INFO org.terracotta.console - Available Max Runtime Memory: 13653MB
2018-01-03 19:39:13,849 [main] INFO com.tc.objectserver.impl.DistributedObjectServer - Creating server nodeID: NodeID[mainhost:8090]
2018-01-03 19:39:13,884 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Initializing org.terracotta.client.message.tracker.OOOMessageHandlerProvider@61dd025
2018-01-03 19:39:13,897 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Initializing org.ehcache.clustered.server.state.EhcacheStateServiceProvider@4445629
2018-01-03 19:39:13,955 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Initializing org.terracotta.management.service.monitoring.MonitoringServiceProvider@4a22f9e2
2018-01-03 19:39:13,955 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Registering implementation-provided service com.tc.services.PlatformServiceProvider@470f1802
2018-01-03 19:39:13,959 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Registering implementation-provided service com.tc.services.EntityMessengerProvider@3e92efc3
2018-01-03 19:39:13,960 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Registering implementation-provided service com.tc.services.CommunicatorService@2e4b8173
2018-01-03 19:39:13,961 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Initializing com.tc.objectserver.persistence.NullPlatformStorageServiceProvider@34b7ac2f
2018-01-03 19:39:13,964 [main] INFO com.tc.services.TerracottaServiceProviderRegistryImpl - Registering implementation-provided service com.tc.services.LocalMonitoringProducer@6a400542
2018-01-03 19:39:14,004 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 4 worker comm threads for L2_L1
2018-01-03 19:39:14,017 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: TSA Server - HealthChecker Started
2018-01-03 19:39:14,018 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener not requested
2018-01-03 19:39:14,083 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 4 worker comm threads for L2_L2
2018-01-03 19:39:14,087 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: TCGroupManager - HealthChecker Started
2018-01-03 19:39:14,087 [main] INFO com.tc.net.protocol.tcm.CommunicationsManager - HealthCheck CallbackPort Listener not requested
2018-01-03 19:39:14,144 [main] INFO com.tc.objectserver.impl.DistributedObjectServer - This L2 Node ID = NodeID[mainhost:8090]
2018-01-03 19:39:14,145 [main] INFO com.tc.l2.state.StateManagerImpl - Starting election initial state:null
2018-01-03 19:39:14,148 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.net.core.TCConnectionManager - Active connections : 0 out of 0
2018-01-03 19:39:14,151 [WorkerThread(l2_election_handler_stage, 0)] INFO com.tc.l2.state.ElectionManagerImpl - Election Started : Enrollment [ NodeID[mainhost:8090], isNew = true, weights = 0,0,0,0,108,1625753149870297890 ]
2018-01-03 19:39:14,155 [main] INFO com.tc.server.TCServer - Server started as sat-node-b
**** works fine for a while..... and then crashes
ns : 6 out of 9
2018-01-03 19:40:05,875 [WorkerThread(group_events_dispatch_stage, 0)] INFO com.tc.l2.ha.L2HACoordinator - NodeID[alternatehost:8090] joined the cluster
2018-01-03 19:40:05,875 [WorkerThread(group_events_dispatch_stage, 0)] INFO org.terracotta.console - NodeID[alternatehost:8090] joined the cluster
2018-01-03 19:40:05,880 [WorkerThread(l2_state_message_handler_stage, 0)] INFO com.tc.l2.state.StateManagerImpl - Forcing Abort Election for L2StateMessage [ NodeID[alternatehost:8090], type = START_ELECTION, Enrollment [ NodeID[alternatehost:8090], isNew = true, weights = 0,0,0,0,342,-1814468041751002912 ], State[ START-STATE ]] with L2StateMessage [ NodeID[NULL-ID], type = ABORT_ELECTION, Enrollment [ NodeID[mainhost:8090], isNew = false, weights = 9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807 ], State[ ACTIVE-COORDINATOR ]]
2018-01-03 19:40:05,901 [WorkerThread(passive_replication_ack_stage, 0)] INFO com.tc.objectserver.handler.ReplicationSender - Created batch context for passive NodeID[alternatehost:8090] with max batch size 64 and ideal messages in flight 1
2018-01-03 19:40:05,906 [WorkerThread(passive_replication_ack_stage, 0)] INFO com.tc.objectserver.entity.PassiveReplicationBroker - Starting sync to NodeID[alternatehost:8090]
2018-01-03 19:40:06,172 [L2_L1:TCComm Main Selector Thread_R (listen mainhost:8090)] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(6.45e6063ae4334438a3059d150918dfa5.dbd951da-e412-45db-a9b1-e3a7295ddfaf-160be7aa79c.Permanent)[]: Attaching new connection to transport: com.tc.net.core.TCConnectionImpl@164261624: connected: true, closed: false local=mainhost:8090 remote=10.153.180.4:60732 connect=[Wed Jan 03 19:40:06 EST 2018] idle=8ms [0 read, 0 write]
2018-01-03 19:40:06,176 [L2_L1:TCWorkerComm # 2_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. TSA Server - Health monitoring agent started for tappsw-vlan972.sat.cbp.dhs.gov:60732
2018-01-03 19:40:06,179 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Handling client handshake for ClientID[6]
2018-01-03 19:40:06,179 [WorkerThread(client_handshake_stage, 0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Sending handshake acknowledgement to ClientID[6]
2018-01-03 19:40:07,039 [WorkerThread(request_processor_stage, 1, 1)] INFO org.ehcache.clustered.server.store.ClusterTierActiveEntity - Sync started for concurrency key 1.
2018-01-03 19:40:07,053 [WorkerThread(request_processor_stage, 1, 1)] ERROR com.tc.objectserver.entity.ManagedEntityImpl - caught exception during invoke
java.lang.ClassCastException: org.ehcache.clustered.server.internal.messages.EhcacheStateRepoSyncMessage cannot be cast to org.ehcache.clustered.server.internal.messages.EhcacheStateRepoSyncMessage
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_131]
at org.ehcache.clustered.server.store.ClusterTierActiveEntity.synchronizeKeyToPassive(ClusterTierActiveEntity.java:603) ~[ehcache-server-3.4.0.jar:3.4.0 45a1f99e0d65f13bdd379e60a5ca9a7b6d3a90bc]
at com.tc.objectserver.entity.ManagedEntityImpl.performSync(ManagedEntityImpl.java:730) [dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.ManagedEntityImpl.invoke(ManagedEntityImpl.java:513) [dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.ManagedEntityImpl.lambda$sync$12(ManagedEntityImpl.java:946) [dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.invoke(RequestProcessor.java:171) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.run(RequestProcessor.java:162) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:27) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:23) ~[dso-l2-5.3.2.jar:na]
at com.tc.async.impl.AbstractStageQueueImpl$HandledContext.runWithHandler(AbstractStageQueueImpl.java:207) ~[common-5.3.2.jar:na]
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:197) ~[common-5.3.2.jar:na]
2018-01-03 19:40:07,060 [WorkerThread(request_processor_stage, 1, 1)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(request_processor_stage, 1, 1),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
com.tc.exception.TCRuntimeException: Uncaught exception in stage
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:211) ~[common-5.3.2.jar:na]
Caused by: java.lang.RuntimeException: java.lang.ClassCastException: org.ehcache.clustered.server.internal.messages.EhcacheStateRepoSyncMessage cannot be cast to org.ehcache.clustered.server.internal.messages.EhcacheStateRepoSyncMessage
at com.tc.objectserver.entity.ManagedEntityImpl.invoke(ManagedEntityImpl.java:534) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.ManagedEntityImpl.lambda$sync$12(ManagedEntityImpl.java:946) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.invoke(RequestProcessor.java:171) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.run(RequestProcessor.java:162) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:27) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:23) ~[dso-l2-5.3.2.jar:na]
at com.tc.async.impl.AbstractStageQueueImpl$HandledContext.runWithHandler(AbstractStageQueueImpl.java:207) ~[common-5.3.2.jar:na]
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:197) ~[common-5.3.2.jar:na]
Caused by: java.lang.ClassCastException: org.ehcache.clustered.server.internal.messages.EhcacheStateRepoSyncMessage cannot be cast to org.ehcache.clustered.server.internal.messages.EhcacheStateRepoSyncMessage
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_131]
at org.ehcache.clustered.server.store.ClusterTierActiveEntity.synchronizeKeyToPassive(ClusterTierActiveEntity.java:603) ~[ehcache-server-3.4.0.jar:3.4.0 45a1f99e0d65f13bdd379e60a5ca9a7b6d3a90bc]
at com.tc.objectserver.entity.ManagedEntityImpl.performSync(ManagedEntityImpl.java:730) ~[dso-l2-5.3.2.jar:na]
at com.tc.objectserver.entity.ManagedEntityImpl.invoke(ManagedEntityImpl.java:513) ~[dso-l2-5.3.2.jar:na]
... 7 common frames omitted
2018-01-03 19:40:07,106 [WorkerThread(request_processor_stage, 1, 1)] ERROR com.tc.logging.ThreadDumpHandler - Wed Jan 03 19:40:07 EST 2018
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode):
"Dump On Timeout Timer" Id=68 TIMED_WAITING on java.util.TaskQueue@3d18b6d0
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"pool-1-thread-1" Id=67 WAITING on java.util.concurrent.Semaphore$NonfairSync@3785b13
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at com.tc.objectserver.entity.BarrierCompletion.waitForCompletion(BarrierCompletion.java:34)
at com.tc.objectserver.entity.ManagedEntityImpl.sync(ManagedEntityImpl.java:949)
at com.tc.objectserver.entity.ActiveToPassiveReplication$1.run(ActiveToPassiveReplication.java:171)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
"WorkerThread(passive_replication_ack_stage, 0)" Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4dd12e3b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(respond_to_request_stage, 0)" Id=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74471621
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(voltron_message_stage, 0)" Id=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c11804f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(active_to_passive_driver_stage, 0)" Id=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4df0f21d
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"Static Member discovery" Id=56 WAITING on com.tc.net.groups.TCGroupMemberDiscoveryStatic@914bd6e
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.tc.net.groups.TCGroupMemberDiscoveryStatic.pauseDiscovery(TCGroupMemberDiscoveryStatic.java:277)
at com.tc.net.groups.TCGroupMemberDiscoveryStatic$1.run(TCGroupMemberDiscoveryStatic.java:182)
at java.lang.Thread.run(Thread.java:748)
"WorkerThread(l2_state_message_handler_stage, 0)" Id=55 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45ae4473
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(client_handshake_stage, 0)" Id=54 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45bdb771
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 3, 3)" Id=53 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f97d5c1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 2, 2)" Id=52 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2344f690
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(request_processor_stage, 1, 1)" Id=51 RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at com.tc.util.runtime.ThreadDumpUtil.getThreadDump(ThreadDumpUtil.java:126)
at com.tc.logging.ThreadDumpHandler.callbackOnExit(ThreadDumpHandler.java:35)
at com.tc.lang.ThrowableHandlerImpl.handleDefaultException(ThrowableHandlerImpl.java:212)
- locked <0x147eb357> (a java.lang.Object)
at com.tc.lang.ThrowableHandlerImpl.handleThrowable(ThrowableHandlerImpl.java:148)
at com.tc.lang.TCThreadGroup.uncaughtException(TCThreadGroup.java:38)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
"WorkerThread(request_processor_stage, 0, 0)" Id=50 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3c508896
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_discovery_stage, 0)" Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d586b00
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_handshake_message_stage, 0)" Id=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@92cec4d
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(receive_group_message_stage, 0)" Id=47 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68291234
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(server_entity_response_message_stage, 0)" Id=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5085339f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(l2_state_change_stage, 0)" Id=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e4b42e4
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(l2_election_handler_stage, 0)" Id=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@70eb8be4
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_events_dispatch_stage, 0)" Id=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50739f7f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 3, 3)" Id=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@29935725
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 2, 2)" Id=41 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d8b6c91
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 1, 1)" Id=40 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41937e6f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(hydrate_message_stage, 0, 0)" Id=39 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d578945
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.MultiStageQueueImpl$MultiSourceQueueImpl.poll(MultiStageQueueImpl.java:445)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(group_hydrate_message_stage, 0)" Id=38 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ececdb5
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"WorkerThread(platform_information_request, 0)" Id=37 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@817c119
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at com.tc.async.impl.SingletonStageQueueImpl$SourceQueueImpl.poll(SingletonStageQueueImpl.java:253)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:194)
"HealthChecker" Id=36 TIMED_WAITING on com.tc.util.concurrent.SetOnceFlag@522da672
at java.lang.Object.wait(Native Method)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:245)
at java.lang.Thread.run(Thread.java:748)
"L2_L2:TCComm Main Selector Thread_W (listen 0.0.0.0:9430)" Id=35 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x630110d3> (a sun.nio.ch.Util$3)
- locked <0x2beada63> (a java.util.Collections$UnmodifiableSet)
- locked <0x1d43c90e> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCComm Main Selector Thread_R (listen 0.0.0.0:9430)" Id=34 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2262f2e2> (a sun.nio.ch.Util$3)
- locked <0x498ee9f5> (a java.util.Collections$UnmodifiableSet)
- locked <0x34ca0b65> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 3_W" Id=33 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x3742af4b> (a sun.nio.ch.Util$3)
- locked <0x202bea3> (a java.util.Collections$UnmodifiableSet)
- locked <0x3c7d2e57> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 3_R" Id=32 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x58089b9e> (a sun.nio.ch.Util$3)
- locked <0x7a124f23> (a java.util.Collections$UnmodifiableSet)
- locked <0x6d720164> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 2_W" Id=31 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x38a8950b> (a sun.nio.ch.Util$3)
- locked <0x24e03deb> (a java.util.Collections$UnmodifiableSet)
- locked <0x4ce30503> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 2_R" Id=30 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x316754da> (a sun.nio.ch.Util$3)
- locked <0x4d9b8f30> (a java.util.Collections$UnmodifiableSet)
- locked <0x68b1ec0d> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 1_W" Id=29 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x3c2e3705> (a sun.nio.ch.Util$3)
- locked <0x376475de> (a java.util.Collections$UnmodifiableSet)
- locked <0x26178e0d> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 1_R" Id=28 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x4121df46> (a sun.nio.ch.Util$3)
- locked <0xcfa4e78> (a java.util.Collections$UnmodifiableSet)
- locked <0x18a410bc> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 0_W" Id=27 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2599243> (a sun.nio.ch.Util$3)
- locked <0x4d4db6a1> (a java.util.Collections$UnmodifiableSet)
- locked <0x139ab8fe> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L2:TCWorkerComm # 0_R" Id=26 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2b836bc1> (a sun.nio.ch.Util$3)
- locked <0x65062f78> (a java.util.Collections$UnmodifiableSet)
- locked <0x45de65c7> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"TC Group Manager Handshake timer" Id=25 TIMED_WAITING on java.util.TaskQueue@34578ca
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"Reconnect timer" Id=24 WAITING on java.util.TaskQueue@33a8d600
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)
"HealthChecker" Id=23 TIMED_WAITING on com.tc.util.concurrent.SetOnceFlag@678152e2
at java.lang.Object.wait(Native Method)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:245)
at java.lang.Thread.run(Thread.java:748)
"L2_L1:TCComm Main Selector Thread_W (listen mainhost:8090)" Id=22 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x51c7e388> (a sun.nio.ch.Util$3)
- locked <0x79ec4252> (a java.util.Collections$UnmodifiableSet)
- locked <0xc0193b9> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCComm Main Selector Thread_R (listen mainhost:8090)" Id=21 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x61fac699> (a sun.nio.ch.Util$3)
- locked <0xf4fa28d> (a java.util.Collections$UnmodifiableSet)
- locked <0x142a9fe4> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 3_W" Id=20 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x14b65a83> (a sun.nio.ch.Util$3)
- locked <0x234f9882> (a java.util.Collections$UnmodifiableSet)
- locked <0x51e60dac> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 3_R" Id=19 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x11737dcb> (a sun.nio.ch.Util$3)
- locked <0x37b74f24> (a java.util.Collections$UnmodifiableSet)
- locked <0x68634ad8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 2_W" Id=18 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x909a115> (a sun.nio.ch.Util$3)
- locked <0x39e08da> (a java.util.Collections$UnmodifiableSet)
- locked <0xeddb9e2> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 2_R" Id=17 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x7c9a076b> (a sun.nio.ch.Util$3)
- locked <0x385f00e4> (a java.util.Collections$UnmodifiableSet)
- locked <0x4bf417ed> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 1_W" Id=16 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2805b5d8> (a sun.nio.ch.Util$3)
- locked <0x4084eec> (a java.util.Collections$UnmodifiableSet)
- locked <0x441026a9> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 1_R" Id=15 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2f300ab9> (a sun.nio.ch.Util$3)
- locked <0x2a9ec1f2> (a java.util.Collections$UnmodifiableSet)
- locked <0x6c278563> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 0_W" Id=14 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x2947c320> (a sun.nio.ch.Util$3)
- locked <0x60d6173> (a java.util.Collections$UnmodifiableSet)
- locked <0x5ac0a111> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"L2_L1:TCWorkerComm # 0_R" Id=13 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xa9c3bb7> (a sun.nio.ch.Util$3)
- locked <0x1dd5a707> (a java.util.Collections$UnmodifiableSet)
- locked <0x70be5b37> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:602)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:344)
"SampledCounterManager Timer" Id=11 TIMED_WAITING on java.util.TaskQueue@2f29b418
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"SingleThreadedTimer" Id=10 WAITING on com.tc.services.SingleThreadedTimer@448373c9
at java.lang.Object.wait(Native Method)
at com.tc.services.SingleThreadedTimer.getNextRunnable(SingleThreadedTimer.java:165)
at com.tc.services.SingleThreadedTimer.access$000(SingleThreadedTimer.java:33)
at com.tc.services.SingleThreadedTimer$1.run(SingleThreadedTimer.java:38)
"Signal Dispatcher" Id=4 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@74de19c1
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@763b938e
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" Id=1 WAITING on com.tc.server.TCServerImpl@14d07025
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.tc.server.TCServerImpl.waitUntilShutdown(TCServerImpl.java:457)
at com.tc.server.TCServerMain.main(TCServerMain.java:76)
2018-01-03 19:40:07,313 [WorkerThread(request_processor_stage, 1, 1)] INFO org.terracotta.dump - [dump] {
className=com.tc.async.impl.StageManagerImpl
monitor=false
stages={
platform_information_request=0
group_hydrate_message_stage=0
hydrate_message_stage=0
passive_replication_ack_stage=0
group_events_dispatch_stage=0
l2_election_handler_stage=0
l2_state_change_stage=0
active_to_passive_driver_stage=0
server_entity_response_message_stage=0
receive_group_message_stage=0
respond_to_request_stage=0
group_handshake_message_stage=0
group_discovery_stage=0
passive_outgoing_response_stage=0
voltron_message_stage=0
request_processor_stage=0
client_handshake_stage=0
l2_state_message_handler_stage=0
passive_replication_stage=0
}
}
{
className=com.tc.objectserver.persistence.Persistor
started=true
clusterState={
StripeID=StripeID[45e6063ae4334438a3059d150918dfa5]
Version=5.3.2.0.0
Initial State=null
Current State=State[ ACTIVE-COORDINATOR ]
isDBClean=true
}
entityPersistor={
className=com.tc.objectserver.persistence.EntityPersistor
existingEntities=[
org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity-tvs$BIO_UI_CACHE
org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity-tvs$HEALTH_CHECK
org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity-tvs
org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity-tvs$BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient-VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs
org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity-tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
org.terracotta.management.entity.nms.agent.client.NmsAgentEntity-NmsAgent
org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity-tvs$BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
]
journals={
ClientID[0]=[
JournalEntry{operation=CREATE, transactionID=15}
]
ClientID[1]=[
JournalEntry{operation=CREATE, transactionID=11}
]
ClientID[2]=[
JournalEntry{operation=CREATE, transactionID=11}
]
ClientID[3]=[
JournalEntry{operation=CREATE, transactionID=11}
]
ClientID[4]=[
JournalEntry{operation=CREATE, transactionID=11}
]
ClientID[5]=[
JournalEntry{operation=CREATE, transactionID=11}
]
ClientID[6]=[
JournalEntry{operation=CREATE, transactionID=1}
]
}
nextConsumerID=22
}
clientPersistor={
className=com.tc.objectserver.persistence.ClientStatePersistor
clients=[
ClientID[0]
ClientID[1]
ClientID[2]
ClientID[3]
ClientID[4]
ClientID[5]
ClientID[6]
]
next=7
}
orderPersistor={
className=com.tc.objectserver.persistence.TransactionOrderPersistor
receivedTransactions=295
permanentClients={
ClientID[0]=[
Global seq Id = 289, local seq id = 61
Global seq Id = 290, local seq id = 62
]
ClientID[1]=[
Global seq Id = 286, local seq id = 95
]
ClientID[2]=[
Global seq Id = 294, local seq id = 54
Global seq Id = 295, local seq id = 53
]
ClientID[3]=[
Global seq Id = 141, local seq id = 28
Global seq Id = 142, local seq id = 29
]
ClientID[4]=[
Global seq Id = 242, local seq id = 32
]
ClientID[5]=[
Global seq Id = 243, local seq id = 22
]
ClientID[6]=[
Global seq Id = 291, local seq id = 1
]
}
regularClients={
}
}
}
{
className=com.tc.net.groups.TCGroupManagerImpl
channelMap={
ChannelID=[0]:mainhost:9430 <--> alternatehost:38792=NodeID[alternatehost:8090]
}
members={
NodeID[alternatehost:8090]=Group Member: NodeID[mainhost:8090] <-> NodeID[alternatehost:8090] ChannelID=[0]:mainhost:9430 <--> alternatehost:38792; Ready:true; Joined: true; memberAdding:false; HighPri: true
}
zapped=[
]
}
{
className=com.tc.l2.ha.ReplicatedClusterStateManagerImpl
state={
connections=[
ConnectionID(2.45e6063ae4334438a3059d150918dfa5.71f8bc5a-9592-4228-9a45-d4bb360adeed-160be77808d.Permanent)[]
ConnectionID(5.45e6063ae4334438a3059d150918dfa5.dbd951da-e412-45db-a9b1-e3a7295ddfaf-160be7aa79c.Permanent)[]
ConnectionID(6.45e6063ae4334438a3059d150918dfa5.dbd951da-e412-45db-a9b1-e3a7295ddfaf-160be7aa79c.Permanent)[]
ConnectionID(0.45e6063ae4334438a3059d150918dfa5.71f8bc5a-9592-4228-9a45-d4bb360adeed-160be77808d.Permanent)[]
ConnectionID(3.45e6063ae4334438a3059d150918dfa5.71f8bc5a-9592-4228-9a45-d4bb360adeed-160be77808d.Permanent)[]
ConnectionID(4.45e6063ae4334438a3059d150918dfa5.71f8bc5a-9592-4228-9a45-d4bb360adeed-160be77808d.Permanent)[]
ConnectionID(1.45e6063ae4334438a3059d150918dfa5.71f8bc5a-9592-4228-9a45-d4bb360adeed-160be77808d.Permanent)[]
]
nextChannelID=7
currentState=State[ ACTIVE-COORDINATOR ]
stripeID=StripeID[45e6063ae4334438a3059d150918dfa5]
}
stateManager=StateManagerImpl:State[ ACTIVE-COORDINATOR ]
}
{
className=com.tc.objectserver.entity.EntityManagerImpl
size=9
entities=[
{
id=EntityID{className='platform', entityName='root'}
}
{
fetchID=FetchID=[1]
entityID=EntityID{className='org.terracotta.management.entity.nms.agent.client.NmsAgentEntity', entityName='NmsAgent'}
consumerID=1
referenceCount=-1
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
instance=org.terracotta.management.entity.nms.agent.server.ActiveNmsAgentServerEntity@79d74522
clientCount=6
clients=[
ClientDescriptorImpl{nodeID=ClientID[5], entityDescriptor=ClientInstanceID(5)}
ClientDescriptorImpl{nodeID=ClientID[4], entityDescriptor=ClientInstanceID(5)}
ClientDescriptorImpl{nodeID=ClientID[1], entityDescriptor=ClientInstanceID(5)}
ClientDescriptorImpl{nodeID=ClientID[0], entityDescriptor=ClientInstanceID(6)}
ClientDescriptorImpl{nodeID=ClientID[3], entityDescriptor=ClientInstanceID(5)}
ClientDescriptorImpl{nodeID=ClientID[2], entityDescriptor=ClientInstanceID(5)}
]
}
}
{
fetchID=FetchID=[17]
entityID=EntityID{className='org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity', entityName='tvs$HEALTH_CHECK'}
consumerID=17
referenceCount=2
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
managerIdentifier=tvs
storeIdentifier=HEALTH_CHECK
configuration={
storedKeyType=java.lang.String
storedValueType=gov.cbp.ace.st.common.cache.CacheContainer
keySerializerType=org.ehcache.impl.serialization.StringSerializer
valueSerializerType=org.ehcache.impl.serialization.CompactJavaSerializer
consistency=EVENTUAL
pool={
resourceName=primary-server-resource
size=2097152
}
}
clientCount=2
clients=[
{
attached=true
storeIdentifier=HEALTH_CHECK
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[4], entityDescriptor=ClientInstanceID(4)}
clientIdentifier=3d5ea6ce-d564-4295-ab12-9c9ce69d356d
}
{
attached=true
storeIdentifier=HEALTH_CHECK
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[5], entityDescriptor=ClientInstanceID(4)}
clientIdentifier=b5f0fa10-f5bc-4547-afd0-49b3d265c33b
}
]
}
}
{
fetchID=FetchID=[3]
entityID=EntityID{className='org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity', entityName='tvs'}
consumerID=3
referenceCount=6
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
managerIdentifier=tvs
configuration={
defaultServerResource=null
pools={
}
}
clientCount=6
clients=[
{
attached=true
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[3], entityDescriptor=ClientInstanceID(3)}
clientIdentifier=04e34107-b12c-4e34-b610-d15038ddabc6
}
{
attached=true
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[2], entityDescriptor=ClientInstanceID(3)}
clientIdentifier=ceb0c0cf-ec33-4069-8436-5cad2173fa86
}
{
attached=true
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[1], entityDescriptor=ClientInstanceID(3)}
clientIdentifier=05b4ff77-93a0-4bf5-9254-b0037824b0f4
}
{
attached=true
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[0], entityDescriptor=ClientInstanceID(4)}
clientIdentifier=bc45a618-5ce3-4336-ae7c-b4c92116ee2b
}
{
attached=true
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[5], entityDescriptor=ClientInstanceID(3)}
clientIdentifier=b5f0fa10-f5bc-4547-afd0-49b3d265c33b
}
{
attached=true
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[4], entityDescriptor=ClientInstanceID(3)}
clientIdentifier=3d5ea6ce-d564-4295-ab12-9c9ce69d356d
}
]
}
}
{
fetchID=FetchID=[4]
entityID=EntityID{className='org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient', entityName='VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs'}
consumerID=4
referenceCount=6
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
holders={
shared=[ClientDescriptorImpl{nodeID=ClientID[2], entityDescriptor=ClientInstanceID(2)}, ClientDescriptorImpl{nodeID=ClientID[3], entityDescriptor=ClientInstanceID(2)}, ClientDescriptorImpl{nodeID=ClientID[0], entityDescriptor=ClientInstanceID(3)}, ClientDescriptorImpl{nodeID=ClientID[1], entityDescriptor=ClientInstanceID(2)}, ClientDescriptorImpl{nodeID=ClientID[4], entityDescriptor=ClientInstanceID(2)}, ClientDescriptorImpl{nodeID=ClientID[5], entityDescriptor=ClientInstanceID(2)}]
}
releaseListeners=[]
}
}
{
fetchID=FetchID=[5]
entityID=EntityID{className='org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity', entityName='tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE'}
consumerID=5
referenceCount=1
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
managerIdentifier=tvs
storeIdentifier=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
configuration={
storedKeyType=java.lang.String
storedValueType=gov.cbp.ace.st.common.cache.CacheContainer
keySerializerType=org.ehcache.impl.serialization.StringSerializer
valueSerializerType=org.ehcache.impl.serialization.CompactJavaSerializer
consistency=EVENTUAL
pool={
resourceName=primary-server-resource
size=2097152
}
}
clientCount=1
clients=[
{
attached=true
storeIdentifier=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[0], entityDescriptor=ClientInstanceID(5)}
clientIdentifier=bc45a618-5ce3-4336-ae7c-b4c92116ee2b
}
]
}
}
{
fetchID=FetchID=[8]
entityID=EntityID{className='org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity', entityName='tvs$BIO_UI_CACHE'}
consumerID=8
referenceCount=1
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
managerIdentifier=tvs
storeIdentifier=BIO_UI_CACHE
configuration={
storedKeyType=java.lang.String
storedValueType=gov.cbp.ace.st.common.cache.CacheContainer
keySerializerType=org.ehcache.impl.serialization.StringSerializer
valueSerializerType=org.ehcache.impl.serialization.CompactJavaSerializer
consistency=EVENTUAL
pool={
resourceName=primary-server-resource
size=2097152
}
}
clientCount=1
clients=[
{
attached=true
storeIdentifier=BIO_UI_CACHE
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[1], entityDescriptor=ClientInstanceID(4)}
clientIdentifier=05b4ff77-93a0-4bf5-9254-b0037824b0f4
}
]
}
}
{
fetchID=FetchID=[11]
entityID=EntityID{className='org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity', entityName='tvs$BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE'}
consumerID=11
referenceCount=1
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
managerIdentifier=tvs
storeIdentifier=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
configuration={
storedKeyType=java.lang.String
storedValueType=gov.cbp.ace.st.common.cache.CacheContainer
keySerializerType=org.ehcache.impl.serialization.StringSerializer
valueSerializerType=org.ehcache.impl.serialization.CompactJavaSerializer
consistency=EVENTUAL
pool={
resourceName=primary-server-resource
size=2097152
}
}
clientCount=1
clients=[
{
attached=true
storeIdentifier=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[2], entityDescriptor=ClientInstanceID(4)}
clientIdentifier=ceb0c0cf-ec33-4069-8436-5cad2173fa86
}
]
}
}
{
fetchID=FetchID=[14]
entityID=EntityID{className='org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity', entityName='tvs$BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE'}
consumerID=14
referenceCount=1
waitForExclusive={
deferring=false
queue=[
]
}
retirement={
running={
}
waitingForDeferredRegistration={
}
mostRecentRegisteredToKey={
}
}
entityState={
managerIdentifier=tvs
storeIdentifier=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
configuration={
storedKeyType=java.lang.String
storedValueType=gov.cbp.ace.st.common.cache.CacheContainer
keySerializerType=org.ehcache.impl.serialization.StringSerializer
valueSerializerType=org.ehcache.impl.serialization.CompactJavaSerializer
consistency=EVENTUAL
pool={
resourceName=primary-server-resource
size=2097152
}
}
clientCount=1
clients=[
{
attached=true
storeIdentifier=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
clientDescriptor=ClientDescriptorImpl{nodeID=ClientID[3], entityDescriptor=ClientInstanceID(4)}
clientIdentifier=04e34107-b12c-4e34-b610-d15038ddabc6
}
]
}
}
]
}
{
services=[
{
org.terracotta.client.message.tracker.OOOMessageHandlerProvider={
tvs###BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE={
com.tc.objectserver.entity.ClientSourceIdImpl@0={
TrackedResponses=[61, 62]
}
}
tvs###BIO_UI_CACHE={
com.tc.objectserver.entity.ClientSourceIdImpl@1={
TrackedResponses=[95]
}
}
tvs###BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE={
com.tc.objectserver.entity.ClientSourceIdImpl@2={
TrackedResponses=[53, 54]
}
}
tvs###HEALTH_CHECK={
com.tc.objectserver.entity.ClientSourceIdImpl@4={
TrackedResponses=[]
}
com.tc.objectserver.entity.ClientSourceIdImpl@5={
TrackedResponses=[]
}
}
tvs###BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE={
com.tc.objectserver.entity.ClientSourceIdImpl@3={
TrackedResponses=[]
}
}
}
}
{
org.ehcache.clustered.server.state.EhcacheStateServiceProvider={
tvs={
defaultServerResource=null
configured=true
pools={
}
stores=[BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE, BIO_UI_CACHE, BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE, HEALTH_CHECK, BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE]
}
}
}
{
org.terracotta.management.service.monitoring.MonitoringServiceProvider={
cluster={
stripes=[
{
id=SINGLE
name=SINGLE
servers=[
{
id=sat-node-a
serverEntities=[
{
id=NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity
type=org.terracotta.management.entity.nms.agent.client.NmsAgentEntity
name=NmsAgent
consumerId=1
managementRegistry=null
}
{
id=VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient
type=org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient
name=VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs
consumerId=4
managementRegistry=null
}
{
id=tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
type=org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
name=tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
consumerId=5
managementRegistry={
contextContainer={
consumerId=5
subContexts=[
]
}
capabilities=[
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=5
alias=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
type=Pool
serverResource=primary-server-resource
size=2097152
allocationType=dedicated
}
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Pool:AllocatedSize
type=SIZE
}
]
}
{
name=ServerStoreSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=5
alias=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
type=ServerStore
resourcePoolType=dedicated
allocatedMemoryAtTime=32768
tableCapacityAtTime=2048
vitalMemoryAtTime=32768
longSizeAtTime=0
dataAllocatedMemoryAtTime=0
dataOccupiedMemoryAtTime=0
dataSizeAtTime=0
dataVitalMemoryAtTime=0
resourcePoolDedicatedResourceName=primary-server-resource
resourcePoolDedicatedSize=2097152
}
{
type=ServerStoreSettings
clusterTierManager=tvs
time=1515026407017
}
]
}
{
name=ServerStoreStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Store:AllocatedMemory
type=SIZE
}
{
name=Store:DataAllocatedMemory
type=SIZE
}
{
name=Store:DataOccupiedMemory
type=SIZE
}
{
name=Store:DataSize
type=SIZE
}
{
name=Store:DataVitalMemory
type=SIZE
}
{
name=Store:Entries
type=COUNTER
}
{
name=Store:OccupiedMemory
type=SIZE
}
{
name=Store:RemovedSlotCount
type=COUNTER
}
{
name=Store:TableCapacity
type=SIZE
}
{
name=Store:UsedSlotCount
type=COUNTER
}
{
name=Store:VitalMemory
type=SIZE
}
]
}
]
}
}
{
id=tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity
type=org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity
name=tvs
consumerId=3
managementRegistry={
contextContainer={
consumerId=3
subContexts=[
]
}
capabilities=[
{
name=ClusterTierManagerSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=3
alias=tvs
type=ClusterTierManager
defaultServerResource=null
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
]
}
]
}
}
]
serverName=sat-node-a
hostName=alternatehost
hostAddress=alternatehost
bindAddress=alternatehost
bindPort=8090
groupPort=9430
state=SYNCHRONIZING
version=5.3.2
buildId=2017-08-25 at 14:11:55 PDT (Revision 221ca10a1ccc60b459d50f4d332d86930752b79c from UNKNOWN)
startTime=1515026404542
upTimeSec=1
activateTime=1515026405895
}
{
id=sat-node-b
serverEntities=[
{
id=NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity
type=org.terracotta.management.entity.nms.agent.client.NmsAgentEntity
name=NmsAgent
consumerId=1
managementRegistry=null
}
{
id=VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient
type=org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient
name=VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs
consumerId=4
managementRegistry=null
}
{
id=tvs$BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
type=org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
name=tvs$BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
consumerId=11
managementRegistry={
contextContainer={
consumerId=11
subContexts=[
]
}
capabilities=[
{
name=ClusterTierClientStateSettings
context=[
{
name=consumerId
required=true
}
{
name=clientId
required=true
}
{
name=type
required=true
}
]
descriptors=[
{
consumerId=11
alias=46...@10.153.180.4:Ehcache:tvs:e2dd8828387e43a6a25bd51df7fbf1d3
type=ClusterTierClientState
attached=true
store=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=11
alias=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
type=Pool
serverResource=primary-server-resource
size=2097152
allocationType=dedicated
}
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Pool:AllocatedSize
type=SIZE
}
]
}
{
name=ServerStoreSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=11
alias=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
type=ServerStore
resourcePoolType=dedicated
allocatedMemoryAtTime=32768
tableCapacityAtTime=2048
vitalMemoryAtTime=32768
longSizeAtTime=0
dataAllocatedMemoryAtTime=0
dataOccupiedMemoryAtTime=0
dataSizeAtTime=0
dataVitalMemoryAtTime=0
resourcePoolDedicatedResourceName=primary-server-resource
resourcePoolDedicatedSize=2097152
}
{
type=ServerStoreSettings
clusterTierManager=tvs
time=1515026371807
}
]
}
{
name=ServerStoreStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Store:AllocatedMemory
type=SIZE
}
{
name=Store:DataAllocatedMemory
type=SIZE
}
{
name=Store:DataOccupiedMemory
type=SIZE
}
{
name=Store:DataSize
type=SIZE
}
{
name=Store:DataVitalMemory
type=SIZE
}
{
name=Store:Entries
type=COUNTER
}
{
name=Store:OccupiedMemory
type=SIZE
}
{
name=Store:RemovedSlotCount
type=COUNTER
}
{
name=Store:TableCapacity
type=SIZE
}
{
name=Store:UsedSlotCount
type=COUNTER
}
{
name=Store:VitalMemory
type=SIZE
}
]
}
]
}
}
{
id=tvs$BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
type=org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
name=tvs$BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
consumerId=14
managementRegistry={
contextContainer={
consumerId=14
subContexts=[
]
}
capabilities=[
{
name=ClusterTierClientStateSettings
context=[
{
name=consumerId
required=true
}
{
name=clientId
required=true
}
{
name=type
required=true
}
]
descriptors=[
{
consumerId=14
alias=46...@10.153.180.4:Ehcache:tvs:3ffec250cc1941e69060f74710b6c695
type=ClusterTierClientState
attached=true
store=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=14
alias=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
type=Pool
serverResource=primary-server-resource
size=2097152
allocationType=dedicated
}
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Pool:AllocatedSize
type=SIZE
}
]
}
{
name=ServerStoreSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=14
alias=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
type=ServerStore
resourcePoolType=dedicated
allocatedMemoryAtTime=32768
tableCapacityAtTime=2048
vitalMemoryAtTime=32768
longSizeAtTime=0
dataAllocatedMemoryAtTime=0
dataOccupiedMemoryAtTime=0
dataSizeAtTime=0
dataVitalMemoryAtTime=0
resourcePoolDedicatedResourceName=primary-server-resource
resourcePoolDedicatedSize=2097152
}
{
type=ServerStoreSettings
clusterTierManager=tvs
time=1515026375449
}
]
}
{
name=ServerStoreStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Store:AllocatedMemory
type=SIZE
}
{
name=Store:DataAllocatedMemory
type=SIZE
}
{
name=Store:DataOccupiedMemory
type=SIZE
}
{
name=Store:DataSize
type=SIZE
}
{
name=Store:DataVitalMemory
type=SIZE
}
{
name=Store:Entries
type=COUNTER
}
{
name=Store:OccupiedMemory
type=SIZE
}
{
name=Store:RemovedSlotCount
type=COUNTER
}
{
name=Store:TableCapacity
type=SIZE
}
{
name=Store:UsedSlotCount
type=COUNTER
}
{
name=Store:VitalMemory
type=SIZE
}
]
}
]
}
}
{
id=tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
type=org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
name=tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
consumerId=5
managementRegistry={
contextContainer={
consumerId=5
subContexts=[
]
}
capabilities=[
{
name=ClusterTierClientStateSettings
context=[
{
name=consumerId
required=true
}
{
name=clientId
required=true
}
{
name=type
required=true
}
]
descriptors=[
{
consumerId=5
alias=46...@10.153.180.4:Ehcache:tvs:8d69e2fa172e4a25af5064dc55601b62
type=ClusterTierClientState
attached=true
store=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=5
alias=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
type=Pool
serverResource=primary-server-resource
size=2097152
allocationType=dedicated
}
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Pool:AllocatedSize
type=SIZE
}
]
}
{
name=ServerStoreSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=5
alias=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
type=ServerStore
resourcePoolType=dedicated
allocatedMemoryAtTime=32768
tableCapacityAtTime=2048
vitalMemoryAtTime=32768
longSizeAtTime=0
dataAllocatedMemoryAtTime=0
dataOccupiedMemoryAtTime=0
dataSizeAtTime=0
dataVitalMemoryAtTime=0
resourcePoolDedicatedResourceName=primary-server-resource
resourcePoolDedicatedSize=2097152
}
{
type=ServerStoreSettings
clusterTierManager=tvs
time=1515026371157
}
]
}
{
name=ServerStoreStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Store:AllocatedMemory
type=SIZE
}
{
name=Store:DataAllocatedMemory
type=SIZE
}
{
name=Store:DataOccupiedMemory
type=SIZE
}
{
name=Store:DataSize
type=SIZE
}
{
name=Store:DataVitalMemory
type=SIZE
}
{
name=Store:Entries
type=COUNTER
}
{
name=Store:OccupiedMemory
type=SIZE
}
{
name=Store:RemovedSlotCount
type=COUNTER
}
{
name=Store:TableCapacity
type=SIZE
}
{
name=Store:UsedSlotCount
type=COUNTER
}
{
name=Store:VitalMemory
type=SIZE
}
]
}
]
}
}
{
id=tvs$BIO_UI_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
type=org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
name=tvs$BIO_UI_CACHE
consumerId=8
managementRegistry={
contextContainer={
consumerId=8
subContexts=[
]
}
capabilities=[
{
name=ClusterTierClientStateSettings
context=[
{
name=consumerId
required=true
}
{
name=clientId
required=true
}
{
name=type
required=true
}
]
descriptors=[
{
consumerId=8
alias=46...@10.153.180.4:Ehcache:tvs:437393391bf34789a8b2b4b0a44957d6
type=ClusterTierClientState
attached=true
store=BIO_UI_CACHE
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=8
alias=BIO_UI_CACHE
type=Pool
serverResource=primary-server-resource
size=2097152
allocationType=dedicated
}
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Pool:AllocatedSize
type=SIZE
}
]
}
{
name=ServerStoreSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=8
alias=BIO_UI_CACHE
type=ServerStore
resourcePoolType=dedicated
allocatedMemoryAtTime=32768
tableCapacityAtTime=2048
vitalMemoryAtTime=32768
longSizeAtTime=0
dataAllocatedMemoryAtTime=0
dataOccupiedMemoryAtTime=0
dataSizeAtTime=0
dataVitalMemoryAtTime=0
resourcePoolDedicatedResourceName=primary-server-resource
resourcePoolDedicatedSize=2097152
}
{
type=ServerStoreSettings
clusterTierManager=tvs
time=1515026371489
}
]
}
{
name=ServerStoreStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Store:AllocatedMemory
type=SIZE
}
{
name=Store:DataAllocatedMemory
type=SIZE
}
{
name=Store:DataOccupiedMemory
type=SIZE
}
{
name=Store:DataSize
type=SIZE
}
{
name=Store:DataVitalMemory
type=SIZE
}
{
name=Store:Entries
type=COUNTER
}
{
name=Store:OccupiedMemory
type=SIZE
}
{
name=Store:RemovedSlotCount
type=COUNTER
}
{
name=Store:TableCapacity
type=SIZE
}
{
name=Store:UsedSlotCount
type=COUNTER
}
{
name=Store:VitalMemory
type=SIZE
}
]
}
]
}
}
{
id=tvs$HEALTH_CHECK:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
type=org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity
name=tvs$HEALTH_CHECK
consumerId=17
managementRegistry={
contextContainer={
consumerId=17
subContexts=[
]
}
capabilities=[
{
name=ClusterTierClientStateSettings
context=[
{
name=consumerId
required=true
}
{
name=clientId
required=true
}
{
name=type
required=true
}
]
descriptors=[
{
consumerId=17
alias=46...@10.153.180.4:Ehcache:tvs:cbc856de35cb4dfa99b9bc91ac043744
type=ClusterTierClientState
attached=true
store=HEALTH_CHECK
}
{
consumerId=17
alias=63...@10.153.180.4:Ehcache:tvs:d21c7c7dcebc4c1380f1e868bd45de01
type=ClusterTierClientState
attached=true
store=HEALTH_CHECK
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=17
alias=HEALTH_CHECK
type=Pool
serverResource=primary-server-resource
size=2097152
allocationType=dedicated
}
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Pool:AllocatedSize
type=SIZE
}
]
}
{
name=ServerStoreSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=17
alias=HEALTH_CHECK
type=ServerStore
resourcePoolType=dedicated
allocatedMemoryAtTime=40960
tableCapacityAtTime=2048
vitalMemoryAtTime=33312
longSizeAtTime=2
dataAllocatedMemoryAtTime=8192
dataOccupiedMemoryAtTime=544
dataSizeAtTime=8192
dataVitalMemoryAtTime=544
resourcePoolDedicatedResourceName=primary-server-resource
resourcePoolDedicatedSize=2097152
}
{
type=ServerStoreSettings
clusterTierManager=tvs
time=1515026400112
}
]
}
{
name=ServerStoreStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
name=Store:AllocatedMemory
type=SIZE
}
{
name=Store:DataAllocatedMemory
type=SIZE
}
{
name=Store:DataOccupiedMemory
type=SIZE
}
{
name=Store:DataSize
type=SIZE
}
{
name=Store:DataVitalMemory
type=SIZE
}
{
name=Store:Entries
type=COUNTER
}
{
name=Store:OccupiedMemory
type=SIZE
}
{
name=Store:RemovedSlotCount
type=COUNTER
}
{
name=Store:TableCapacity
type=SIZE
}
{
name=Store:UsedSlotCount
type=COUNTER
}
{
name=Store:VitalMemory
type=SIZE
}
]
}
]
}
}
{
id=tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity
type=org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity
name=tvs
consumerId=3
managementRegistry={
contextContainer={
consumerId=3
subContexts=[
]
}
capabilities=[
{
name=ClientStateSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=3
alias=46...@10.153.180.4:Ehcache:tvs:8d69e2fa172e4a25af5064dc55601b62
type=ClientState
attached=true
}
{
consumerId=3
alias=46...@10.153.180.4:Ehcache:tvs:437393391bf34789a8b2b4b0a44957d6
type=ClientState
attached=true
}
{
consumerId=3
alias=46...@10.153.180.4:Ehcache:tvs:e2dd8828387e43a6a25bd51df7fbf1d3
type=ClientState
attached=true
}
{
consumerId=3
alias=46...@10.153.180.4:Ehcache:tvs:3ffec250cc1941e69060f74710b6c695
type=ClientState
attached=true
}
{
consumerId=3
alias=46...@10.153.180.4:Ehcache:tvs:cbc856de35cb4dfa99b9bc91ac043744
type=ClientState
attached=true
}
{
consumerId=3
alias=63...@10.153.180.4:Ehcache:tvs:d21c7c7dcebc4c1380f1e868bd45de01
type=ClientState
attached=true
}
]
}
{
name=ClusterTierManagerSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
{
consumerId=3
alias=tvs
type=ClusterTierManager
defaultServerResource=null
}
]
}
{
name=PoolSettings
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
]
}
{
name=PoolStatistics
context=[
{
name=consumerId
required=true
}
{
name=type
required=true
}
{
name=alias
required=true
}
]
descriptors=[
]
}
]
}
}
]
serverName=sat-node-b
hostName=mainhost
hostAddress=mainhost
bindAddress=mainhost
bindPort=8090
groupPort=9430
state=ACTIVE
version=5.3.2
buildId=2017-08-25 at 14:11:55 PDT (Revision 221ca10a1ccc60b459d50f4d332d86930752b79c from UNKNOWN)
startTime=1515026353671
upTimeSec=5
activateTime=1515026353671
}
]
}
]
clients=[
{
id=46...@10.153.180.4:Ehcache:tvs:3ffec250cc1941e69060f74710b6c695
pid=46036
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=3ffec250cc1941e69060f74710b6c695
vmId=46...@10.153.180.4
clientId=46...@10.153.180.4:Ehcache:tvs:3ffec250cc1941e69060f74710b6c695
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[caching, client]
connections=[
{
id=3ffec250cc1941e69060f74710b6c695:SINGLE:sat-node-b:10.153.180.4:36790
logicalConnectionUid=3ffec250cc1941e69060f74710b6c695
clientEndpoint={
address=10.153.180.4
port=36790
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity=1
VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient=1
tvs$BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity=1
tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity=1
}
}
]
managementRegistry={
contextContainer={
cacheManagerName=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
subContexts=[
{
cacheName=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
subContexts=[
]
}
]
}
capabilities=[
{
name=ActionsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=clear
returnType=void
parameters=[
]
}
{
name=get
returnType=java.lang.Object
parameters=[
{
name=key
type=java.lang.Object
}
]
}
{
name=put
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
{
name=value
type=java.lang.Object
}
]
}
{
name=remove
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
]
}
]
}
{
name=DiagnosticCalls
context=[
]
descriptors=[
{
name=getThreadDump
returnType=java.lang.String
parameters=[
]
}
]
}
{
name=NmsAgentService
context=[
]
descriptors=[
]
}
{
name=SettingsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
cacheName=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
keyType=java.lang.String
valueType=gov.cbp.ace.st.common.cache.CacheContainer
resourcePools={
clustered-dedicated={
level=10
persistent=true
type=MEMORY
size=2
unit=MB
serverResource=primary-server-resource
clusterTier=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
}
}
}
{
cacheManagerDescription=caches:
BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE:
keyType: java.lang.String
valueType: gov.cbp.ace.st.common.cache.CacheContainer
serviceConfigurations:
- org.ehcache.clustered.client.config.ClusteredStoreConfiguration
evictionAdvisor: None
expiry: TimeToLiveExpiry
resourcePools:
pools:
clustered-dedicated:
size: 2 MB (persistent)
tierHeight: 10
services:
- org.ehcache.clustered.client.config.ClusteringServiceConfiguration:
clusterUri: terracotta://bio-cache-s.sat.cbp.dhs.gov:8090/tvs
readOperationTimeout: TimeoutDuration{2 SECONDS}
autoCreate: true
defaultServerResource: null
resourcePools:
status=AVAILABLE
managementContext={
cacheManagerName=BIO_TRVLR_DATA_TASK_GALLERY_PERSON_IDS_CACHE
}
tags=[Ljava.lang.String;@15f1e69b
}
]
}
{
name=StatisticCollectorCapability
context=[
{
name=cacheManagerName
required=true
}
]
descriptors=[
{
name=isRunning
returnType=boolean
parameters=[
]
}
{
name=startStatisticCollector
returnType=void
parameters=[
{
name=interval
type=long
}
{
name=unit
type=java.util.concurrent.TimeUnit
}
]
}
{
name=stopStatisticCollector
returnType=void
parameters=[
]
}
]
}
{
name=StatisticsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=Cache:EvictionCount
type=COUNTER
}
{
name=Cache:ExpirationCount
type=COUNTER
}
{
name=Cache:HitCount
type=COUNTER
}
{
name=Cache:MissCount
type=COUNTER
}
{
name=Cache:PutCount
type=COUNTER
}
{
name=Cache:RemovalCount
type=COUNTER
}
{
name=Cache:UpdateCount
type=COUNTER
}
{
name=Clustered:AllocatedByteSize
type=SIZE
}
{
name=Clustered:EvictionCount
type=COUNTER
}
{
name=Clustered:ExpirationCount
type=COUNTER
}
{
name=Clustered:HitCount
type=COUNTER
}
{
name=Clustered:MappingCount
type=COUNTER
}
{
name=Clustered:MaxMappingCount
type=COUNTER
}
{
name=Clustered:MissCount
type=COUNTER
}
{
name=Clustered:OccupiedByteSize
type=SIZE
}
{
name=Clustered:PutCount
type=COUNTER
}
{
name=Clustered:RemovalCount
type=COUNTER
}
{
name=Clustered:UpdateCount
type=COUNTER
}
]
}
]
}
}
{
id=46...@10.153.180.4:Ehcache:tvs:437393391bf34789a8b2b4b0a44957d6
pid=46036
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=437393391bf34789a8b2b4b0a44957d6
vmId=46...@10.153.180.4
clientId=46...@10.153.180.4:Ehcache:tvs:437393391bf34789a8b2b4b0a44957d6
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[caching, client]
connections=[
{
id=437393391bf34789a8b2b4b0a44957d6:SINGLE:sat-node-b:10.153.180.4:36746
logicalConnectionUid=437393391bf34789a8b2b4b0a44957d6
clientEndpoint={
address=10.153.180.4
port=36746
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity=1
VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient=1
tvs$BIO_UI_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity=1
tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity=1
}
}
]
managementRegistry={
contextContainer={
cacheManagerName=BIO_UI_CACHE
subContexts=[
{
cacheName=BIO_UI_CACHE
subContexts=[
]
}
]
}
capabilities=[
{
name=ActionsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=clear
returnType=void
parameters=[
]
}
{
name=get
returnType=java.lang.Object
parameters=[
{
name=key
type=java.lang.Object
}
]
}
{
name=put
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
{
name=value
type=java.lang.Object
}
]
}
{
name=remove
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
]
}
]
}
{
name=DiagnosticCalls
context=[
]
descriptors=[
{
name=getThreadDump
returnType=java.lang.String
parameters=[
]
}
]
}
{
name=NmsAgentService
context=[
]
descriptors=[
]
}
{
name=SettingsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
cacheName=BIO_UI_CACHE
keyType=java.lang.String
valueType=gov.cbp.ace.st.common.cache.CacheContainer
resourcePools={
clustered-dedicated={
level=10
persistent=true
type=MEMORY
size=2
unit=MB
serverResource=primary-server-resource
clusterTier=BIO_UI_CACHE
}
}
}
{
cacheManagerDescription=caches:
BIO_UI_CACHE:
keyType: java.lang.String
valueType: gov.cbp.ace.st.common.cache.CacheContainer
serviceConfigurations:
- org.ehcache.clustered.client.config.ClusteredStoreConfiguration
evictionAdvisor: None
expiry: TimeToLiveExpiry
resourcePools:
pools:
clustered-dedicated:
size: 2 MB (persistent)
tierHeight: 10
services:
- org.ehcache.clustered.client.config.ClusteringServiceConfiguration:
clusterUri: terracotta://bio-cache-s.sat.cbp.dhs.gov:8090/tvs
readOperationTimeout: TimeoutDuration{2 SECONDS}
autoCreate: true
defaultServerResource: null
resourcePools:
status=AVAILABLE
managementContext={
cacheManagerName=BIO_UI_CACHE
}
tags=[Ljava.lang.String;@778c504c
}
]
}
{
name=StatisticCollectorCapability
context=[
{
name=cacheManagerName
required=true
}
]
descriptors=[
{
name=isRunning
returnType=boolean
parameters=[
]
}
{
name=startStatisticCollector
returnType=void
parameters=[
{
name=interval
type=long
}
{
name=unit
type=java.util.concurrent.TimeUnit
}
]
}
{
name=stopStatisticCollector
returnType=void
parameters=[
]
}
]
}
{
name=StatisticsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=Cache:EvictionCount
type=COUNTER
}
{
name=Cache:ExpirationCount
type=COUNTER
}
{
name=Cache:HitCount
type=COUNTER
}
{
name=Cache:MissCount
type=COUNTER
}
{
name=Cache:PutCount
type=COUNTER
}
{
name=Cache:RemovalCount
type=COUNTER
}
{
name=Cache:UpdateCount
type=COUNTER
}
{
name=Clustered:AllocatedByteSize
type=SIZE
}
{
name=Clustered:EvictionCount
type=COUNTER
}
{
name=Clustered:ExpirationCount
type=COUNTER
}
{
name=Clustered:HitCount
type=COUNTER
}
{
name=Clustered:MappingCount
type=COUNTER
}
{
name=Clustered:MaxMappingCount
type=COUNTER
}
{
name=Clustered:MissCount
type=COUNTER
}
{
name=Clustered:OccupiedByteSize
type=SIZE
}
{
name=Clustered:PutCount
type=COUNTER
}
{
name=Clustered:RemovalCount
type=COUNTER
}
{
name=Clustered:UpdateCount
type=COUNTER
}
]
}
]
}
}
{
id=46...@10.153.180.4:Ehcache:tvs:8d69e2fa172e4a25af5064dc55601b62
pid=46036
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=8d69e2fa172e4a25af5064dc55601b62
vmId=46...@10.153.180.4
clientId=46...@10.153.180.4:Ehcache:tvs:8d69e2fa172e4a25af5064dc55601b62
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[caching, client]
connections=[
{
id=8d69e2fa172e4a25af5064dc55601b62:SINGLE:sat-node-b:10.153.180.4:36732
logicalConnectionUid=8d69e2fa172e4a25af5064dc55601b62
clientEndpoint={
address=10.153.180.4
port=36732
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity=1
VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient=1
tvs$BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity=1
tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity=1
}
}
]
managementRegistry={
contextContainer={
cacheManagerName=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
subContexts=[
{
cacheName=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
subContexts=[
]
}
]
}
capabilities=[
{
name=ActionsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=clear
returnType=void
parameters=[
]
}
{
name=get
returnType=java.lang.Object
parameters=[
{
name=key
type=java.lang.Object
}
]
}
{
name=put
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
{
name=value
type=java.lang.Object
}
]
}
{
name=remove
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
]
}
]
}
{
name=DiagnosticCalls
context=[
]
descriptors=[
{
name=getThreadDump
returnType=java.lang.String
parameters=[
]
}
]
}
{
name=NmsAgentService
context=[
]
descriptors=[
]
}
{
name=SettingsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
cacheName=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
keyType=java.lang.String
valueType=gov.cbp.ace.st.common.cache.CacheContainer
resourcePools={
clustered-dedicated={
level=10
persistent=true
type=MEMORY
size=2
unit=MB
serverResource=primary-server-resource
clusterTier=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
}
}
}
{
cacheManagerDescription=caches:
BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE:
keyType: java.lang.String
valueType: gov.cbp.ace.st.common.cache.CacheContainer
serviceConfigurations:
- org.ehcache.clustered.client.config.ClusteredStoreConfiguration
evictionAdvisor: None
expiry: TimeToLiveExpiry
resourcePools:
pools:
clustered-dedicated:
size: 2 MB (persistent)
tierHeight: 10
services:
- org.ehcache.clustered.client.config.ClusteringServiceConfiguration:
clusterUri: terracotta://bio-cache-s.sat.cbp.dhs.gov:8090/tvs
readOperationTimeout: TimeoutDuration{2 SECONDS}
autoCreate: true
defaultServerResource: null
resourcePools:
status=AVAILABLE
managementContext={
cacheManagerName=BIO_TRVLR_DATA_TASK_UNAUGMENTED_CACHE
}
tags=[Ljava.lang.String;@1046111b
}
]
}
{
name=StatisticCollectorCapability
context=[
{
name=cacheManagerName
required=true
}
]
descriptors=[
{
name=isRunning
returnType=boolean
parameters=[
]
}
{
name=startStatisticCollector
returnType=void
parameters=[
{
name=interval
type=long
}
{
name=unit
type=java.util.concurrent.TimeUnit
}
]
}
{
name=stopStatisticCollector
returnType=void
parameters=[
]
}
]
}
{
name=StatisticsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=Cache:EvictionCount
type=COUNTER
}
{
name=Cache:ExpirationCount
type=COUNTER
}
{
name=Cache:HitCount
type=COUNTER
}
{
name=Cache:MissCount
type=COUNTER
}
{
name=Cache:PutCount
type=COUNTER
}
{
name=Cache:RemovalCount
type=COUNTER
}
{
name=Cache:UpdateCount
type=COUNTER
}
{
name=Clustered:AllocatedByteSize
type=SIZE
}
{
name=Clustered:EvictionCount
type=COUNTER
}
{
name=Clustered:ExpirationCount
type=COUNTER
}
{
name=Clustered:HitCount
type=COUNTER
}
{
name=Clustered:MappingCount
type=COUNTER
}
{
name=Clustered:MaxMappingCount
type=COUNTER
}
{
name=Clustered:MissCount
type=COUNTER
}
{
name=Clustered:OccupiedByteSize
type=SIZE
}
{
name=Clustered:PutCount
type=COUNTER
}
{
name=Clustered:RemovalCount
type=COUNTER
}
{
name=Clustered:UpdateCount
type=COUNTER
}
]
}
]
}
}
{
id=46...@10.153.180.4:Ehcache:tvs:cbc856de35cb4dfa99b9bc91ac043744
pid=46036
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=cbc856de35cb4dfa99b9bc91ac043744
vmId=46...@10.153.180.4
clientId=46...@10.153.180.4:Ehcache:tvs:cbc856de35cb4dfa99b9bc91ac043744
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[caching, client]
connections=[
{
id=cbc856de35cb4dfa99b9bc91ac043744:SINGLE:sat-node-b:10.153.180.4:36978
logicalConnectionUid=cbc856de35cb4dfa99b9bc91ac043744
clientEndpoint={
address=10.153.180.4
port=36978
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity=1
VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient=1
tvs$HEALTH_CHECK:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity=1
tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity=1
}
}
]
managementRegistry={
contextContainer={
cacheManagerName=HEALTH_CHECK
subContexts=[
{
cacheName=HEALTH_CHECK
subContexts=[
]
}
]
}
capabilities=[
{
name=ActionsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=clear
returnType=void
parameters=[
]
}
{
name=get
returnType=java.lang.Object
parameters=[
{
name=key
type=java.lang.Object
}
]
}
{
name=put
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
{
name=value
type=java.lang.Object
}
]
}
{
name=remove
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
]
}
]
}
{
name=DiagnosticCalls
context=[
]
descriptors=[
{
name=getThreadDump
returnType=java.lang.String
parameters=[
]
}
]
}
{
name=NmsAgentService
context=[
]
descriptors=[
]
}
{
name=SettingsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
cacheName=HEALTH_CHECK
keyType=java.lang.String
valueType=gov.cbp.ace.st.common.cache.CacheContainer
resourcePools={
clustered-dedicated={
level=10
persistent=true
type=MEMORY
size=2
unit=MB
serverResource=primary-server-resource
clusterTier=HEALTH_CHECK
}
}
}
{
cacheManagerDescription=caches:
HEALTH_CHECK:
keyType: java.lang.String
valueType: gov.cbp.ace.st.common.cache.CacheContainer
serviceConfigurations:
- org.ehcache.clustered.client.config.ClusteredStoreConfiguration
evictionAdvisor: None
expiry: TimeToLiveExpiry
resourcePools:
pools:
clustered-dedicated:
size: 2 MB (persistent)
tierHeight: 10
services:
- org.ehcache.clustered.client.config.ClusteringServiceConfiguration:
clusterUri: terracotta://bio-cache-s.sat.cbp.dhs.gov:8090/tvs
readOperationTimeout: TimeoutDuration{2 SECONDS}
autoCreate: true
defaultServerResource: null
resourcePools:
status=AVAILABLE
managementContext={
cacheManagerName=HEALTH_CHECK
}
tags=[Ljava.lang.String;@63ad9dc4
}
]
}
{
name=StatisticCollectorCapability
context=[
{
name=cacheManagerName
required=true
}
]
descriptors=[
{
name=isRunning
returnType=boolean
parameters=[
]
}
{
name=startStatisticCollector
returnType=void
parameters=[
{
name=interval
type=long
}
{
name=unit
type=java.util.concurrent.TimeUnit
}
]
}
{
name=stopStatisticCollector
returnType=void
parameters=[
]
}
]
}
{
name=StatisticsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=Cache:EvictionCount
type=COUNTER
}
{
name=Cache:ExpirationCount
type=COUNTER
}
{
name=Cache:HitCount
type=COUNTER
}
{
name=Cache:MissCount
type=COUNTER
}
{
name=Cache:PutCount
type=COUNTER
}
{
name=Cache:RemovalCount
type=COUNTER
}
{
name=Cache:UpdateCount
type=COUNTER
}
{
name=Clustered:AllocatedByteSize
type=SIZE
}
{
name=Clustered:EvictionCount
type=COUNTER
}
{
name=Clustered:ExpirationCount
type=COUNTER
}
{
name=Clustered:HitCount
type=COUNTER
}
{
name=Clustered:MappingCount
type=COUNTER
}
{
name=Clustered:MaxMappingCount
type=COUNTER
}
{
name=Clustered:MissCount
type=COUNTER
}
{
name=Clustered:OccupiedByteSize
type=SIZE
}
{
name=Clustered:PutCount
type=COUNTER
}
{
name=Clustered:RemovalCount
type=COUNTER
}
{
name=Clustered:UpdateCount
type=COUNTER
}
]
}
]
}
}
{
id=46...@10.153.180.4:Ehcache:tvs:e2dd8828387e43a6a25bd51df7fbf1d3
pid=46036
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=e2dd8828387e43a6a25bd51df7fbf1d3
vmId=46...@10.153.180.4
clientId=46...@10.153.180.4:Ehcache:tvs:e2dd8828387e43a6a25bd51df7fbf1d3
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[caching, client]
connections=[
{
id=e2dd8828387e43a6a25bd51df7fbf1d3:SINGLE:sat-node-b:10.153.180.4:36758
logicalConnectionUid=e2dd8828387e43a6a25bd51df7fbf1d3
clientEndpoint={
address=10.153.180.4
port=36758
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity=1
VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient=1
tvs$BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity=1
tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity=1
}
}
]
managementRegistry={
contextContainer={
cacheManagerName=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
subContexts=[
{
cacheName=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
subContexts=[
]
}
]
}
capabilities=[
{
name=ActionsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=clear
returnType=void
parameters=[
]
}
{
name=get
returnType=java.lang.Object
parameters=[
{
name=key
type=java.lang.Object
}
]
}
{
name=put
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
{
name=value
type=java.lang.Object
}
]
}
{
name=remove
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
]
}
]
}
{
name=DiagnosticCalls
context=[
]
descriptors=[
{
name=getThreadDump
returnType=java.lang.String
parameters=[
]
}
]
}
{
name=NmsAgentService
context=[
]
descriptors=[
]
}
{
name=SettingsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
cacheName=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
keyType=java.lang.String
valueType=gov.cbp.ace.st.common.cache.CacheContainer
resourcePools={
clustered-dedicated={
level=10
persistent=true
type=MEMORY
size=2
unit=MB
serverResource=primary-server-resource
clusterTier=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
}
}
}
{
cacheManagerDescription=caches:
BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE:
keyType: java.lang.String
valueType: gov.cbp.ace.st.common.cache.CacheContainer
serviceConfigurations:
- org.ehcache.clustered.client.config.ClusteredStoreConfiguration
evictionAdvisor: None
expiry: TimeToLiveExpiry
resourcePools:
pools:
clustered-dedicated:
size: 2 MB (persistent)
tierHeight: 10
services:
- org.ehcache.clustered.client.config.ClusteringServiceConfiguration:
clusterUri: terracotta://bio-cache-s.sat.cbp.dhs.gov:8090/tvs
readOperationTimeout: TimeoutDuration{2 SECONDS}
autoCreate: true
defaultServerResource: null
resourcePools:
status=AVAILABLE
managementContext={
cacheManagerName=BIO_TRVLR_DATA_TASK_AUGMENTED_CACHE
}
tags=[Ljava.lang.String;@79f26a14
}
]
}
{
name=StatisticCollectorCapability
context=[
{
name=cacheManagerName
required=true
}
]
descriptors=[
{
name=isRunning
returnType=boolean
parameters=[
]
}
{
name=startStatisticCollector
returnType=void
parameters=[
{
name=interval
type=long
}
{
name=unit
type=java.util.concurrent.TimeUnit
}
]
}
{
name=stopStatisticCollector
returnType=void
parameters=[
]
}
]
}
{
name=StatisticsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=Cache:EvictionCount
type=COUNTER
}
{
name=Cache:ExpirationCount
type=COUNTER
}
{
name=Cache:HitCount
type=COUNTER
}
{
name=Cache:MissCount
type=COUNTER
}
{
name=Cache:PutCount
type=COUNTER
}
{
name=Cache:RemovalCount
type=COUNTER
}
{
name=Cache:UpdateCount
type=COUNTER
}
{
name=Clustered:AllocatedByteSize
type=SIZE
}
{
name=Clustered:EvictionCount
type=COUNTER
}
{
name=Clustered:ExpirationCount
type=COUNTER
}
{
name=Clustered:HitCount
type=COUNTER
}
{
name=Clustered:MappingCount
type=COUNTER
}
{
name=Clustered:MaxMappingCount
type=COUNTER
}
{
name=Clustered:MissCount
type=COUNTER
}
{
name=Clustered:OccupiedByteSize
type=SIZE
}
{
name=Clustered:PutCount
type=COUNTER
}
{
name=Clustered:RemovalCount
type=COUNTER
}
{
name=Clustered:UpdateCount
type=COUNTER
}
]
}
]
}
}
{
id=63...@10.153.180.4:Ehcache:tvs:9731fd545d1f4f1a9aa54b01de4543db
pid=63329
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=9731fd545d1f4f1a9aa54b01de4543db
vmId=63...@10.153.180.4
clientId=63...@10.153.180.4:Ehcache:tvs:9731fd545d1f4f1a9aa54b01de4543db
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[]
connections=[
{
id=9731fd545d1f4f1a9aa54b01de4543db:SINGLE:sat-node-b:10.153.180.4:60732
logicalConnectionUid=9731fd545d1f4f1a9aa54b01de4543db
clientEndpoint={
address=10.153.180.4
port=60732
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
}
}
]
managementRegistry=null
}
{
id=63...@10.153.180.4:Ehcache:tvs:d21c7c7dcebc4c1380f1e868bd45de01
pid=63329
hostAddress=10.153.180.4
name=Ehcache:tvs
logicalConnectionUid=d21c7c7dcebc4c1380f1e868bd45de01
vmId=63...@10.153.180.4
clientId=63...@10.153.180.4:Ehcache:tvs:d21c7c7dcebc4c1380f1e868bd45de01
hostName=tappsw-vlan972.sat.cbp.dhs.gov
tags=[caching, client]
connections=[
{
id=d21c7c7dcebc4c1380f1e868bd45de01:SINGLE:sat-node-b:10.153.180.4:60670
logicalConnectionUid=d21c7c7dcebc4c1380f1e868bd45de01
clientEndpoint={
address=10.153.180.4
port=60670
}
stripeId=SINGLE
serverId=sat-node-b
serverEntityIds={
NmsAgent:org.terracotta.management.entity.nms.agent.client.NmsAgentEntity=1
VoltronReadWriteLock-ClusterTierManagerClientEntityFactory-AccessLock-tvs:org.ehcache.clustered.client.internal.lock.VoltronReadWriteLockClient=1
tvs$HEALTH_CHECK:org.ehcache.clustered.client.internal.store.InternalClusterTierClientEntity=1
tvs:org.ehcache.clustered.client.internal.InternalClusterTierManagerClientEntity=1
}
}
]
managementRegistry={
contextContainer={
cacheManagerName=HEALTH_CHECK
subContexts=[
{
cacheName=HEALTH_CHECK
subContexts=[
]
}
]
}
capabilities=[
{
name=ActionsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=clear
returnType=void
parameters=[
]
}
{
name=get
returnType=java.lang.Object
parameters=[
{
name=key
type=java.lang.Object
}
]
}
{
name=put
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
{
name=value
type=java.lang.Object
}
]
}
{
name=remove
returnType=void
parameters=[
{
name=key
type=java.lang.Object
}
]
}
]
}
{
name=DiagnosticCalls
context=[
]
descriptors=[
{
name=getThreadDump
returnType=java.lang.String
parameters=[
]
}
]
}
{
name=NmsAgentService
context=[
]
descriptors=[
]
}
{
name=SettingsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
cacheName=HEALTH_CHECK
keyType=java.lang.String
valueType=gov.cbp.ace.st.common.cache.CacheContainer
resourcePools={
clustered-dedicated={
level=10
persistent=true
type=MEMORY
size=2
unit=MB
serverResource=primary-server-resource
clusterTier=HEALTH_CHECK
}
}
}
{
cacheManagerDescription=caches:
HEALTH_CHECK:
keyType: java.lang.String
valueType: gov.cbp.ace.st.common.cache.CacheContainer
serviceConfigurations:
- org.ehcache.clustered.client.config.ClusteredStoreConfiguration
evictionAdvisor: None
expiry: TimeToLiveExpiry
resourcePools:
pools:
clustered-dedicated:
size: 2 MB (persistent)
tierHeight: 10
services:
- org.ehcache.clustered.client.config.ClusteringServiceConfiguration:
clusterUri: terracotta://bio-cache-s.sat.cbp.dhs.gov:8090/tvs
readOperationTimeout: TimeoutDuration{2 SECONDS}
autoCreate: true
defaultServerResource: null
resourcePools:
status=AVAILABLE
managementContext={
cacheManagerName=HEALTH_CHECK
}
tags=[Ljava.lang.String;@6c694b56
}
]
}
{
name=StatisticCollectorCapability
context=[
{
name=cacheManagerName
required=true
}
]
descriptors=[
{
name=isRunning
returnType=boolean
parameters=[
]
}
{
name=startStatisticCollector
returnType=void
parameters=[
{
name=interval
type=long
}
{
name=unit
type=java.util.concurrent.TimeUnit
}
]
}
{
name=stopStatisticCollector
returnType=void
parameters=[
]
}
]
}
{
name=StatisticsCapability
context=[
{
name=cacheManagerName
required=true
}
{
name=cacheName
required=true
}
]
descriptors=[
{
name=Cache:EvictionCount
type=COUNTER
}
{
name=Cache:ExpirationCount
type=COUNTER
}
{
name=Cache:HitCount
type=COUNTER
}
{
name=Cache:MissCount
type=COUNTER
}
{
name=Cache:PutCount
type=COUNTER
}
{
name=Cache:RemovalCount
type=COUNTER
}
{
name=Cache:UpdateCount
type=COUNTER
}
{
name=Clustered:AllocatedByteSize
type=SIZE
}
{
name=Clustered:EvictionCount
type=COUNTER
}
{
name=Clustered:ExpirationCount
type=COUNTER
}
{
name=Clustered:HitCount
type=COUNTER
}
{
name=Clustered:MappingCount
type=COUNTER
}
{
name=Clustered:MaxMappingCount
type=COUNTER
}
{
name=Clustered:MissCount
type=COUNTER
}
{
name=Clustered:OccupiedByteSize
type=SIZE
}
{
name=Clustered:PutCount
type=COUNTER
}
{
name=Clustered:RemovalCount
type=COUNTER
}
{
name=Clustered:UpdateCount
type=COUNTER
}
]
}
]
}
}
]
}
}
}
{
com.tc.objectserver.persistence.NullPlatformStorageServiceProvider={
0={
key=entities_alive.map
}
}
}
{
com.tc.services.PlatformServiceProvider=com.tc.services.PlatformServiceProvider@470f1802
}
{
com.tc.services.EntityMessengerProvider=com.tc.services.EntityMessengerProvider@3e92efc3
}
{
com.tc.services.CommunicatorService=com.tc.services.CommunicatorService@2e4b8173
}
{
com.tc.services.LocalMonitoringProducer=com.tc.services.LocalMonitoringProducer@6a400542
}
]
}
{
ExtendedConfigs={
org.terracotta.offheapresource.OffHeapResourcesProvider={
primary-server-resource={
capacity=10737418240
available=10726932480
}
main={
capacity=12884901888
available=12884901888
}
}
}
}
2018-01-03 19:40:07,317 [WorkerThread(request_processor_stage, 1, 1)] INFO com.tc.server.TCServerMain - ExitState : CallbackOnExitState[Throwable: class com.tc.exception.TCRuntimeException; RestartNeeded: false]; AutoRestart: true
2018-01-03 19:40:09,323 [CommonShutDownHook] INFO com.tc.objectserver.impl.DistributedObjectServer - L2 Exiting...
Reply all
Reply to author
Forward
0 new messages