HTTP Post Error After 1M Events Consumed
51 views
Skip to first unread message
Fredrik
May 26, 2016, 9:21:34 PM5/26/16
to CDAP User
Hello,
I've wrote a simple application to consume stream event and store it into dataset. I sent the HBase rows as the JSON events to the stream (HTTP post), but it failed roughly at 1M lines (after 2.5 hours) twice, the flow logs show the message below. What it is about? Why it happened?
However, the YARN scheduler shows fine during this time period, plenty of memory, the flow application kept running.
016-05-26 12:48:29,528 - ERROR [FlowletProcessDriver-updater-
I've wrote a simple application to consume stream event and store it into dataset. I sent the HBase rows as the JSON events to the stream (HTTP post), but it failed roughly at 1M lines (after 2.5 hours) twice, the flow logs show the message below. What it is about? Why it happened?
However, the YARN scheduler shows fine during this time period, plenty of memory, the flow application kept running.
016-05-26 12:48:29,528 - ERROR [FlowletProcessDriver-updater-
0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,526 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,932 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,522 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,928 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,934 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:31,539 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:31,539 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:31,531 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:37,943 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:37,934 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:37,943 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,548 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,548 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.TransactionServiceClient@226] - Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4af1c5c6: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,586 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,547 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,547 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.TransactionServiceClient@226] - Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4e43aa13: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,581 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,686 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:04,091 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,538 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,539 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.TransactionServiceClient@226] - Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@3326b2c0: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,558 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,662 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:04,065 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,694 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:04,098 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:05,694 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:05,667 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:05,702 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:12,110 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@279] - Transaction operation failed: Exception from commit for transaction 1464266909398000000. co.cask.tephra.TransactionFailureException: Exception from commit for transaction 1464266909398000000. at co.cask.tephra.TransactionContext.commit(TransactionContext.java:295) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.finish(TransactionContext.java:116) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$DelayedDiscardingTransactionContext.finish(SingleThreadDatasetCache.java:409) ~[co.cask.cdap.cdap-data-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.postProcess(FlowletProcessDriver.java:272) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:240) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:59) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:187) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.RuntimeException: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4e43aa13: java.net.ConnectException: Connection refused at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:323) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.commit(TransactionContext.java:286) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 11 common frames omitted Caused by: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4e43aa13: java.net.ConnectException: Connection refused at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:228) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:186) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:312) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 12 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:122) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:83) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:46) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:39) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.getOrCreate(ElasticPool.java:136) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.obtain(ElasticPool.java:123) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider.getCloseableClient(PooledClientProvider.java:99) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:215) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 14 common frames omitted Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_77] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_77] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_77] at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] ... 23 common frames omitted
2016-05-26 12:49:12,080 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@279] - Transaction operation failed: Exception from commit for transaction 1464266909401000000. co.cask.tephra.TransactionFailureException: Exception from commit for transaction 1464266909401000000. at co.cask.tephra.TransactionContext.commit(TransactionContext.java:295) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.finish(TransactionContext.java:116) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$DelayedDiscardingTransactionContext.finish(SingleThreadDatasetCache.java:409) ~[co.cask.cdap.cdap-data-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.postProcess(FlowletProcessDriver.java:272) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:240) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:59) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:187) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.RuntimeException: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@3326b2c0: java.net.ConnectException: Connection refused at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:323) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.commit(TransactionContext.java:286) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 11 common frames omitted Caused by: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@3326b2c0: java.net.ConnectException: Connection refused at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:228) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:186) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:312) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 12 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:122) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:83) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:46) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:39) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.getOrCreate(ElasticPool.java:136) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.obtain(ElasticPool.java:123) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider.getCloseableClient(PooledClientProvider.java:99) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:215) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 14 common frames omitted Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_77] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_77] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_77] at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] ... 23 common frames omitted
2016-05-26 12:49:12,118 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@279] - Transaction operation failed: Exception from commit for transaction 1464266909397000000. co.cask.tephra.TransactionFailureException: Exception from commit for transaction 1464266909397000000. at co.cask.tephra.TransactionContext.commit(TransactionContext.java:295) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.finish(TransactionContext.java:116) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$DelayedDiscardingTransactionContext.finish(SingleThreadDatasetCache.java:409) ~[co.cask.cdap.cdap-data-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.postProcess(FlowletProcessDriver.java:272) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:240) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:59) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:187) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.RuntimeException: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4af1c5c6: java.net.ConnectException: Connection refused at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:323) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.commit(TransactionContext.java:286) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 11 common frames omitted Caused by: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4af1c5c6: java.net.ConnectException: Connection refused at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:228) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:186) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:312) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 12 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:122) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:83) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:46) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:39) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.getOrCreate(ElasticPool.java:136) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.obtain(ElasticPool.java:123) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider.getCloseableClient(PooledClientProvider.java:99) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:215) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 14 common frames omitted Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_77] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_77] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_77] at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] ... 23 common frames omitted
2016-05-26 12:48:29,526 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,932 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,522 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,928 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:29,934 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:31,539 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:31,539 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:31,531 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:37,943 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:37,934 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:48:37,943 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,548 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,548 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.TransactionServiceClient@226] - Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4af1c5c6: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,586 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,547 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,547 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.TransactionServiceClient@226] - Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4e43aa13: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,581 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,686 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:04,091 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,538 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,539 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.TransactionServiceClient@226] - Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@3326b2c0: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,558 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,662 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:04,065 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:03,694 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:04,098 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:05,694 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:05,667 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:05,702 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.t.d.AbstractClientProvider@124] - Unable to connect to tx service: java.net.ConnectException: Connection refused
2016-05-26 12:49:12,110 - ERROR [FlowletProcessDriver-counter-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@279] - Transaction operation failed: Exception from commit for transaction 1464266909398000000. co.cask.tephra.TransactionFailureException: Exception from commit for transaction 1464266909398000000. at co.cask.tephra.TransactionContext.commit(TransactionContext.java:295) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.finish(TransactionContext.java:116) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$DelayedDiscardingTransactionContext.finish(SingleThreadDatasetCache.java:409) ~[co.cask.cdap.cdap-data-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.postProcess(FlowletProcessDriver.java:272) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:240) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:59) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:187) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.RuntimeException: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4e43aa13: java.net.ConnectException: Connection refused at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:323) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.commit(TransactionContext.java:286) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 11 common frames omitted Caused by: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4e43aa13: java.net.ConnectException: Connection refused at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:228) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:186) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:312) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 12 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:122) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:83) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:46) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:39) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.getOrCreate(ElasticPool.java:136) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.obtain(ElasticPool.java:123) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider.getCloseableClient(PooledClientProvider.java:99) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:215) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 14 common frames omitted Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_77] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_77] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_77] at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] ... 23 common frames omitted
2016-05-26 12:49:12,080 - ERROR [FlowletProcessDriver-reader-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@279] - Transaction operation failed: Exception from commit for transaction 1464266909401000000. co.cask.tephra.TransactionFailureException: Exception from commit for transaction 1464266909401000000. at co.cask.tephra.TransactionContext.commit(TransactionContext.java:295) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.finish(TransactionContext.java:116) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$DelayedDiscardingTransactionContext.finish(SingleThreadDatasetCache.java:409) ~[co.cask.cdap.cdap-data-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.postProcess(FlowletProcessDriver.java:272) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:240) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:59) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:187) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.RuntimeException: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@3326b2c0: java.net.ConnectException: Connection refused at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:323) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.commit(TransactionContext.java:286) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 11 common frames omitted Caused by: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@3326b2c0: java.net.ConnectException: Connection refused at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:228) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:186) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:312) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 12 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:122) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:83) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:46) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:39) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.getOrCreate(ElasticPool.java:136) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.obtain(ElasticPool.java:123) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider.getCloseableClient(PooledClientProvider.java:99) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:215) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 14 common frames omitted Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_77] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_77] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_77] at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] ... 23 common frames omitted
2016-05-26 12:49:12,118 - ERROR [FlowletProcessDriver-updater-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@279] - Transaction operation failed: Exception from commit for transaction 1464266909397000000. co.cask.tephra.TransactionFailureException: Exception from commit for transaction 1464266909397000000. at co.cask.tephra.TransactionContext.commit(TransactionContext.java:295) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.finish(TransactionContext.java:116) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.cdap.data2.dataset2.SingleThreadDatasetCache$DelayedDiscardingTransactionContext.finish(SingleThreadDatasetCache.java:409) ~[co.cask.cdap.cdap-data-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.postProcess(FlowletProcessDriver.java:272) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:240) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:59) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:187) [co.cask.cdap.cdap-app-fabric-3.4.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.RuntimeException: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4af1c5c6: java.net.ConnectException: Connection refused at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[com.google.guava.guava-13.0.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:323) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.TransactionContext.commit(TransactionContext.java:286) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 11 common frames omitted Caused by: java.lang.Exception: Thrift error for co.cask.tephra.distributed.TransactionServiceClient$5@4af1c5c6: java.net.ConnectException: Connection refused at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:228) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:186) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.commit(TransactionServiceClient.java:312) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 12 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:122) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.AbstractClientProvider.newClient(AbstractClientProvider.java:83) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:46) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider$TxClientPool.create(PooledClientProvider.java:39) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.getOrCreate(ElasticPool.java:136) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.ElasticPool.obtain(ElasticPool.java:123) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.PooledClientProvider.getCloseableClient(PooledClientProvider.java:99) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] at co.cask.tephra.distributed.TransactionServiceClient.execute(TransactionServiceClient.java:215) ~[co.cask.tephra.tephra-core-0.7.1.jar:na] ... 14 common frames omitted Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_77] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_77] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_77] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_77] at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ~[org.apache.thrift.libthrift-0.9.0.jar:0.9.0] ... 23 common frames omitted
Poorna Chandra
May 26, 2016, 10:19:37 PM5/26/16
to Fredrik, CDAP User
Hi Fredrik,
Looks like transaction manager is crashing due to insufficient memory. To confirm you'll need to look into transaction service logs. Can you let us know how much memory transaction manager has been configured with?
Thanks,
Poorna.
--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/556758b9-9c05-42cf-8854-53eb1652095c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Fredrik
May 26, 2016, 11:35:27 PM5/26/16
to CDAP User, frederic...@gmail.com
Hello Poorna,
data.tx.memory.mb = 512 MiB
Is this one?
Or
Thanks.
Best Regards,
Fredrik
data.tx.memory.mb = 512 MiB
Is this one?
Or
| data.tx.bind.address | 0.0.0.0 | cdap-default.xml |
| data.tx.bind.port | 15165 | cdap-default.xml |
| data.tx.client.count | 50 | cdap-default.xml |
| data.tx.client.provider | pool | cdap-default.xml |
| data.tx.discovery.service.name | transaction | cdap-default.xml |
| data.tx.hdfs.user | cdap | cdap-default.xml |
| data.tx.janitor.enable | true | cdap-default.xml |
| data.tx.max.instances | 5 | cdap-site.xml |
| data.tx.memory.mb | 512 | cdap-site.xml |
| data.tx.num.cores | 1 | cdap-site.xml |
| data.tx.num.instances | 1 | cdap-site.xml |
| data.tx.server.io.threads | 2 | cdap-default.xml |
| data.tx.server.threads | 25 | cdap-default.xml |
| data.tx.snapshot.codecs | co.cask.cdap.data2.transaction.snapshot.SnapshotCodecV1, co.cask.cdap.data2.transaction.snapshot.SnapshotCodecV2, co.cask.tephra.snapshot.SnapshotCodecV3, co.cask.tephra.snapshot.SnapshotCodecV4 | cdap-default.xml |
| data.tx.snapshot.dir | /cdap/tx.snapshot | cdap-default.xml |
| data.tx.snapshot.interval | 60 | cdap-default.xml |
| data.tx.snapshot.local.dir | data/tx.snapshot | cdap-default.xml |
| data.tx.snapshot.retain | 10 | cdap-default.xml |
| data.tx.thrift.max.read.buffer | 16777216 | cdap-default.xml |
| data.tx.timeout | 30 | cdap-site.xml |
Best Regards,
Fredrik
Poorna Chandra
May 27, 2016, 1:12:23 AM5/27/16
to Fredrik, CDAP User
Hi Fredrik,
512 MB for transaction manager is not sufficient for your use case. I'd suggest you increase it to at least 2 G. Typically, we suggest 4 G of memory for production use. Depending on your cluster size and the number of applications you are running, you may have to increase the memory allocated to other CDAP services too.
Thanks,
Poorna.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/0502401b-2bb1-41ac-b845-dff9aa6d584a%40googlegroups.com.
Fredrik
May 27, 2016, 2:32:57 AM5/27/16
to CDAP User, frederic...@gmail.com
Hello Poorna,
Thank you for the detailed information.
I will try to enlarge the memory set for tx manager. Is it"data.tx.memory.mb"? It is said to be "Memory in megabytes of the transaction clients.
If the tx manager crashed, how do I know it? From UI or YARN, or other places? Need I restart CDAP or the application?
Thanks & Best Regards,
Fredrik
Thank you for the detailed information.
I will try to enlarge the memory set for tx manager. Is it"data.tx.memory.mb"? It is said to be "Memory in megabytes of the transaction clients.
If the tx manager crashed, how do I know it? From UI or YARN, or other places? Need I restart CDAP or the application?
Thanks & Best Regards,
Fredrik
Poorna Chandra
May 27, 2016, 3:11:40 AM5/27/16
to Fredrik, CDAP User
I will try to enlarge the memory set for tx manager. Is it"data.tx.memory.mb"? It is said to be "Memory in megabytes of the transaction clients.
The documentation is incorrect. It should be "Maximum memory in megabytes that can be used by each transaction service instance". I have filed JIRA https://issues.cask.co/browse/CDAP-6105 to fix this.
Use "data.tx.memory.mb" to increase the memory for transaction manager. After updating the configuration, you'll have to restart CDAP.
If the tx manager crashed, how do I know it? From UI or YARN, or other places? Need I restart CDAP or the application?
Transaction manager logs are available in CDAP UI. Go to "Management" > "Services" > "transaction" > "logs".
Thanks,
Poorna.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/b95425c1-228d-44bb-b5ac-fa076fc36fc4%40googlegroups.com.
Fredrik
May 27, 2016, 7:37:19 AM5/27/16
to CDAP User, frederic...@gmail.com
Hello Poorna,
I have enlarged the tx memory following your guide. Thank you.
Best Regards,
Fredrik
I have enlarged the tx memory following your guide. Thank you.
Best Regards,
Fredrik
Reply all
Reply to author
Forward
0 new messages