Installing CDAP on Cloudera Express 5.5

185 views
Skip to first unread message

kira...@gmail.com

unread,
Dec 2, 2015, 7:05:19 PM12/2/15
to CDAP User
http://docs.cask.co/cdap/3.2.1/en/admin-manual/installation/cloudera/step-by-step-cloudera.html#step-by-step-cloudera-add-service

Using Cloudera Express 5.5 image on Virtual Box.   Downloaded and installed Cask 3.2.1 VM CSD and the Parsel.  Installed Node.

The CDAP instance appears to start correctly but when I go to the CDAP UI on localhost:9999 I get 3-4 of these "Session Timeout" pop divs and nothing shows.  There's nothing in the logs in the Cloudera Console and I've tried restarting the CDAP and the whole Cloudera Manager and it still doesn't appear to work.

Any help on how to get this to work would be appreciated.

Thanks,
Lawrence

Rohit Sinha

unread,
Dec 2, 2015, 7:32:09 PM12/2/15
to CDAP User, kira...@gmail.com
Hello Lawrence,
We do not support CDH 5.5 in our current (3.2.1) release. We will start supporting it in our next release 3.3. 
Will it be possible for you to use Cloudera Express 5.4 ? On CDH 5.4 you will be able to run CDAP 3.2.1 

Thanks,
Rohit

kira...@gmail.com

unread,
Dec 3, 2015, 4:49:42 PM12/3/15
to CDAP User, kira...@gmail.com
I downgraded to 5.4 CDH Express image, redid the install of CSD and Parsel 3.2.0_1 and still have the same problem with the Cask UI.  This time however I do see errors in log for the CDAP Gateway/Router Service.  The gateway service isn't responding.

2015-12-03 13:26:32,178 INFO co.cask.cdap.gateway.router.
RouterMain: Initializing Router...
2015-12-03 13:26:33,325 WARN co.cask.cdap.security.auth.DistributedKeyManager: Not adding ACLs on keys in ZooKeeper as Kerberos is not enabled
2015-12-03 13:26:33,368 INFO co.cask.cdap.security.auth.DistributedKeyManager: ZooKeeper ACLs [31,s{'world,'anyone}
] for keys
2015-12-03 13:26:33,394 INFO co.cask.cdap.gateway.router.NettyRouter: Service to Port Mapping - {gateway=11015}
2015-12-03 13:26:33,395 INFO co.cask.cdap.gateway.router.RouterMain: Router initialized.
2015-12-03 13:26:33,396 INFO co.cask.cdap.gateway.router.RouterMain: Starting Router...
2015-12-03 13:26:33,716 INFO co.cask.cdap.security.zookeeper.SharedResourceCache: Initializing SharedResourceCache.  Checking for parent znode /keys
2015-12-03 13:26:33,771 INFO co.cask.cdap.security.zookeeper.SharedResourceCache: Listing existing children for node /keys
2015-12-03 13:26:33,772 INFO org.apache.twill.internal.zookeeper.LeaderElection: Start leader election on quickstart.cloudera:2181/cdap/cdap/security/auth/leader with guid 4d04a4aa-8afc-490d-b670-aac253900e79
2015-12-03 13:26:33,828 INFO co.cask.cdap.security.auth.DistributedKeyManager: Transitioned to leader
2015-12-03 13:26:33,910 INFO co.cask.cdap.gateway.router.NettyRouter: Starting Netty Router for service gateway on address quickstart.cloudera/127.0.0.1:11015...
2015-12-03 13:26:33,958 INFO co.cask.cdap.gateway.router.NettyRouter: Started Netty Router for service gateway on address /127.0.0.1:11015.
2015-12-03 13:26:33,959 INFO co.cask.cdap.gateway.router.RouterMain: Router started.
2015-12-03 13:26:34,028 INFO co.cask.cdap.security.auth.AbstractKeyManager: Changed current key to KeyIdentifier{keyId=42764328, expiration=1449181593828}
2015-12-03 13:26:34,037 INFO co.cask.cdap.security.zookeeper.SharedResourceCache: Listing existing children for node /keys
2015-12-03 13:26:35,234 ERROR co.cask.cdap.gateway.router.RouterServiceLookup: No discoverable endpoints found for service CacheKey{service=appfabric, host=quickstart.cloudera:11015, firstPathPart=/ping}
2015-12-03 13:26:35,244 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0xba604d3a, /127.0.0.1:33135 => /127.0.0.1:11015]
co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /ping
    at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.getDiscoverable(HttpRequestHandler.java:197) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
    at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.messageReceived(HttpRequestHandler.java:106) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
    at co.cask.cdap.gateway.router.handlers.HttpStatusRequestHandler.messageReceived(HttpStatusRequestHandler.java:65) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
2015-12-03 13:27:50,117 ERROR co.cask.cdap.gateway.router.RouterServiceLookup: No discoverable endpoints found for service CacheKey{service=appfabric, host=quickstart.cloudera:11015, firstPathPart=/v3}
2015-12-03 13:27:50,119 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0x9a5be359, /127.0.0.1:33307 => /127.0.0.1:11015]
co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /v3/namespaces
    at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.getDiscoverable(HttpRequestHandler.java:197) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
    at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.messageReceived(HttpRequestHandler.java:106) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
    at co.cask.cdap.gateway.router.handlers.HttpStatusRequestHandler.messageReceived(HttpStatusRequestHandler.java:65) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
2015-12-03 13:27:50,119 ERROR co.cask.cdap.gateway.router.handlers.HttpRequestHandler: Exception raised in Request Handler [id: 0x8b01e635, /127.0.0.1:33306 => /127.0.0.1:11015]
co.cask.cdap.common.HandlerException: No endpoint strategy found for request : /v3/version
    at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.getDiscoverable(HttpRequestHandler.java:197) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
    at co.cask.cdap.gateway.router.handlers.HttpRequestHandler.messageReceived(HttpRequestHandler.java:106) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]
    at co.cask.cdap.gateway.router.handlers.HttpStatusRequestHandler.messageReceived(HttpStatusRequestHandler.java:65) ~[co.cask.cdap.cdap-gateway-3.2.0.jar:na]

Rohit Sinha

unread,
Dec 3, 2015, 5:15:53 PM12/3/15
to CDAP User, kira...@gmail.com
Hello Lawrence,
It looks the app-fabric service is not starting up for some reason.
Can you look into cdap-master logs to see if it has any errors and also send it here so that we can help you resolve this. 

Thanks, 
Rohit. 

kirakane nix

unread,
Dec 3, 2015, 5:37:25 PM12/3/15
to CDAP User, kira...@gmail.com
Master attempts unsuccessfully to connect to the internal Kafka instance a couple times over 5-8 minutes then shuts down.


2:19:38.301 PM		 INFO co.cask.cdap.data2.datafabric.dataset.service.DatasetService 
Starting DatasetService...
2:19:38.305 PM INFO co.cask.cdap.internal.app.runtime.schedule.DistributedSchedulerService 
Starting scheduler.
2:19:38.305 PM INFO co.cask.cdap.internal.app.services.ApplicationLifecycleService 
Starting ApplicationLifecycleService
2:19:38.325 PM INFO co.cask.cdap.internal.app.services.ProgramLifecycleService 
Starting ProgramLifecycleService
2:19:38.326 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaPublisher 
Update Kafka producer broker list: quickstart.cloudera:9092
2:19:38.431 PM INFO co.cask.http.NettyHttpService 
Starting service on address quickstart.cloudera/127.0.0.1:0...
2:19:38.529 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:38.548 PM INFO co.cask.http.NettyHttpService 
Started service on address /127.0.0.1:58652
2:19:38.551 PM INFO co.cask.cdap.internal.app.services.AppFabricServer 
AppFabric HTTP Service started at /127.0.0.1:58652
2:19:39.742 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:40.007 PM INFO co.cask.cdap.data2.util.hbase.HBaseTableUtil 
Table created 'TableId{namespace=system, tableName=datasets.instance}'
2:19:40.008 PM INFO co.cask.cdap.data2.dataset2.InMemoryDatasetFramework 
Created dataset namespace:system/datasetinstance:datasets.instance of type co.cask.cdap.data2.datafabric.dataset.service.mds.DatasetInstanceMDS
2:19:40.855 PM INFO co.cask.cdap.data2.util.hbase.HBaseTableUtil 
Table created 'TableId{namespace=system, tableName=datasets.type}'
2:19:40.856 PM INFO co.cask.cdap.data2.dataset2.InMemoryDatasetFramework 
Created dataset namespace:system/datasetinstance:datasets.type of type co.cask.cdap.data2.datafabric.dataset.service.mds.DatasetTypeMDS
2:19:41.144 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:41.401 PM WARN co.cask.cdap.internal.app.services.ProgramLifecycleService$RunRecordsCorrectorRunnable 
Unable to complete correcting run records: Service 'DatasetService' is not available. Please wait till it is up and running.
2:19:42.879 PM ERROR co.cask.tephra.distributed.AbstractClientProvider 
Unable to discover tx service.
2:19:42.879 PM ERROR co.cask.tephra.distributed.ThreadLocalClientProvider 
Unable to create new tx client for thread: Unable to discover tx service.
2:19:42.880 PM INFO co.cask.tephra.distributed.RetryWithBackoff 
Sleeping 100 ms before retry.
2:19:42.945 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:42.980 PM INFO co.cask.tephra.distributed.TransactionServiceClient 
Retrying startShort after Thrift error: Unable to discover tx service.
2:19:44.981 PM ERROR co.cask.tephra.distributed.AbstractClientProvider 
Unable to discover tx service.
2:19:44.981 PM ERROR co.cask.tephra.distributed.ThreadLocalClientProvider 
Unable to create new tx client for thread: Unable to discover tx service.
2:19:44.981 PM INFO co.cask.tephra.distributed.RetryWithBackoff 
Sleeping 400 ms before retry.
2:19:45.381 PM INFO co.cask.tephra.distributed.TransactionServiceClient 
Retrying startShort after Thrift error: Unable to discover tx service.
2:19:45.546 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:47.382 PM ERROR co.cask.tephra.distributed.AbstractClientProvider 
Unable to discover tx service.
2:19:47.382 PM ERROR co.cask.tephra.distributed.ThreadLocalClientProvider 
Unable to create new tx client for thread: Unable to discover tx service.
2:19:47.382 PM INFO co.cask.tephra.distributed.RetryWithBackoff 
Sleeping 1600 ms before retry.
2:19:48.983 PM INFO co.cask.tephra.distributed.TransactionServiceClient 
Retrying startShort after Thrift error: Unable to discover tx service.
2:19:49.748 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:50.983 PM ERROR co.cask.tephra.distributed.AbstractClientProvider 
Unable to discover tx service.
2:19:50.984 PM ERROR co.cask.tephra.distributed.ThreadLocalClientProvider 
Unable to create new tx client for thread: Unable to discover tx service.
2:19:50.984 PM INFO co.cask.tephra.distributed.RetryWithBackoff 
Sleeping 6400 ms before retry.
2:19:55.749 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:19:57.384 PM INFO co.cask.tephra.distributed.TransactionServiceClient 
Retrying startShort after Thrift error: Unable to discover tx service.
2:19:59.385 PM ERROR co.cask.tephra.distributed.AbstractClientProvider 
Unable to discover tx service.
2:19:59.385 PM ERROR co.cask.tephra.distributed.ThreadLocalClientProvider 
Unable to create new tx client for thread: Unable to discover tx service.
2:19:59.385 PM INFO co.cask.tephra.distributed.RetryWithBackoff 
Sleeping 25600 ms before retry.
2:20:01.751 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:20:05.966 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0001 is in state RUNNING
2:20:07.753 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:20:13.754 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:20:19.756 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:20:24.986 PM INFO co.cask.tephra.distributed.TransactionServiceClient 
Retrying startShort after Thrift error: Unable to discover tx service.
2:20:24.987 PM INFO co.cask.tephra.distributed.AbstractClientProvider 
Service discovered at quickstart.cloudera:15165
2:20:24.987 PM INFO co.cask.tephra.distributed.AbstractClientProvider 
Attempting to connect to tx service at quickstart.cloudera:15165 with timeout 30000 ms.
2:20:25.008 PM INFO co.cask.tephra.distributed.AbstractClientProvider 
Connected to tx service at quickstart.cloudera:15165
2:20:25.251 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=orderedTable-hbase}, className: co.cask.cdap.data2.dataset2.module.lib.hbase.HBaseTableModule, jarLocation: [local]
2:20:25.292 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=metricsTable-hbase}, className: co.cask.cdap.data2.dataset2.module.lib.hbase.HBaseMetricsTableModule, jarLocation: [local]
2:20:25.312 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=core}, className: co.cask.cdap.data2.dataset2.lib.table.CoreDatasetsModule, jarLocation: [local]
2:20:25.367 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=fileSet}, className: co.cask.cdap.data2.dataset2.lib.file.FileSetModule, jarLocation: [local]
2:20:25.386 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=timePartitionedFileSet}, className: co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetModule, jarLocation: [local]
2:20:25.430 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=partitionedFileSet}, className: co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetModule, jarLocation: [local]
2:20:25.480 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=objectMappedTable}, className: co.cask.cdap.data2.dataset2.lib.table.ObjectMappedTableModule, jarLocation: [local]
2:20:25.511 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=cube}, className: co.cask.cdap.data2.dataset2.lib.table.CubeModule, jarLocation: [local]
2:20:25.553 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=usage}, className: co.cask.cdap.data2.registry.UsageDatasetModule, jarLocation: [local]
2:20:25.580 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=businessMetadata}, className: co.cask.cdap.data2.metadata.dataset.BusinessMetadataDatasetModule, jarLocation: [local]
2:20:25.610 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=lineage}, className: co.cask.cdap.data2.metadata.lineage.LineageDatasetModule, jarLocation: [local]
2:20:25.638 PM INFO co.cask.cdap.data2.datafabric.dataset.type.DatasetTypeManager 
adding module: DatasetModule{namespace=system, module=queueDataset}, className: co.cask.cdap.data2.transaction.queue.hbase.HBaseQueueDatasetModule, jarLocation: [local]
2:20:25.664 PM INFO co.cask.http.NettyHttpService 
Starting service on address quickstart.cloudera/127.0.0.1:0...
2:20:25.667 PM INFO co.cask.cdap.data2.datafabric.dataset.service.DatasetTypeHandler 
Starting DatasetTypeHandler
2:20:25.669 PM INFO co.cask.http.NettyHttpService 
Started service on address /127.0.0.1:36943
2:20:25.671 PM INFO co.cask.cdap.data2.datafabric.dataset.service.DatasetService 
Waiting for dataset.executor service to be discoverable
2:20:25.758 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
















































































2:22:31.798 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:22:37.239 PM INFO org.apache.twill.yarn.YarnTwillController 
Failed to access application master.services application_1449175041089_0001 live node in ZK, resort to polling. Failure reason: KeeperErrorCode = NoNode for /instances/b0f464ec-20ff-42f3-b628-9f119ec2cdd4
2:22:37.800 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:22:40.182 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Exception when fetching message on TopicPartition{topic=log, partition=0}.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67]
	at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67]
	at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_67]
	at kafka.network.BlockingChannel.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
2:22:40.262 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0001 completed. Shutting down controller.
2:22:40.270 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Requesting stop of all consumer threads.
2:22:40.271 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Terminate requested Kafka-Consumer-log-0
2:22:40.272 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Wait for all consumer threads to stop.
2:22:40.273 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
All consumer threads stopped.
2:22:40.274 PM INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService 
Stopping KafkaClientService
2:22:40.275 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Stopping Kafka consumer
2:22:40.276 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Kafka Consumer stopped
2:22:40.279 PM INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService 
KafkaClientService stopped
2:22:40.283 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
master.services was terminated; restarting with back-off
2:22:42.403 PM WARN co.cask.cdap.internal.app.services.ProgramLifecycleService$RunRecordsCorrectorRunnable 
Unable to complete correcting run records: Service 'DatasetService' is not available. Please wait till it is up and running.
2:22:43.423 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of dataset.executor Service to 1
2:22:43.430 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of metrics Service to 1
2:22:43.436 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of transaction Service to 1
2:22:43.441 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of streams Service to 1
2:22:43.446 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of explore.service Service to 1
2:22:43.451 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of log.saver Service to 1
2:22:43.459 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of metrics.processor Service to 1
2:22:43.459 PM INFO co.cask.cdap.data.runtime.main.MasterTwillApplication 
Adding explore runnable.
2:22:43.463 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/core-site.xml
2:22:43.464 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/hdfs-site.xml
2:22:43.476 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/hive-site.xml
2:22:43.484 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/mapred-site.xml
2:22:43.485 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/ssl-client.xml
2:22:43.492 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/yarn-site.xml
2:22:43.801 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:22:49.803 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:22:55.804 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:01.806 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:07.162 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0002 is in state RUNNING
2:23:07.808 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:13.810 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:19.811 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:25.814 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:31.815 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:37.817 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:43.818 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:49.820 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:23:55.822 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:01.828 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:07.830 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:13.831 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:19.832 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:25.835 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:31.836 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:37.838 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:43.841 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:49.844 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:24:55.846 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:01.848 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:07.851 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:13.854 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:19.855 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:25.858 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:26.859 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:27.860 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:28.862 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:34.863 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:39.773 PM INFO org.apache.twill.yarn.YarnTwillController 
Failed to access application master.services application_1449175041089_0002 live node in ZK, resort to polling. Failure reason: KeeperErrorCode = NoNode for /instances/587f7872-928a-4798-adfb-9ae6cd5c601f
2:25:40.865 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:42.681 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Exception when fetching message on TopicPartition{topic=log, partition=0}.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67]
	at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67]
	at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_67]
	at kafka.network.BlockingChannel.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
2:25:42.791 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0002 completed. Shutting down controller.
2:25:42.794 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Requesting stop of all consumer threads.
2:25:42.795 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Terminate requested Kafka-Consumer-log-0
2:25:42.796 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Wait for all consumer threads to stop.
2:25:42.797 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
All consumer threads stopped.
2:25:42.799 PM INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService 
Stopping KafkaClientService
2:25:42.800 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Stopping Kafka consumer
2:25:42.801 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Kafka Consumer stopped
2:25:42.802 PM INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService 
KafkaClientService stopped
2:25:42.803 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
master.services was terminated; restarting with back-off
2:25:43.404 PM WARN co.cask.cdap.internal.app.services.ProgramLifecycleService$RunRecordsCorrectorRunnable 
Unable to complete correcting run records: Service 'DatasetService' is not available. Please wait till it is up and running.
2:25:46.866 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:46.906 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of dataset.executor Service to 1
2:25:46.913 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of metrics Service to 1
2:25:46.919 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of transaction Service to 1
2:25:46.924 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of streams Service to 1
2:25:46.930 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of explore.service Service to 1
2:25:46.936 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of log.saver Service to 1
2:25:46.942 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of metrics.processor Service to 1
2:25:46.943 PM INFO co.cask.cdap.data.runtime.main.MasterTwillApplication 
Adding explore runnable.
2:25:46.946 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/core-site.xml
2:25:46.948 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/hdfs-site.xml
2:25:46.957 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/hive-site.xml
2:25:46.965 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/mapred-site.xml
2:25:46.967 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/ssl-client.xml
2:25:46.973 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/yarn-site.xml
2:25:52.868 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:25:58.869 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:04.871 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:08.601 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0003 is in state RUNNING
2:26:10.873 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:16.875 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:22.877 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:28.880 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:34.882 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:40.883 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:46.885 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:52.887 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:26:58.891 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:04.892 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:10.894 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:16.896 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:22.898 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:28.901 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:34.902 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:40.905 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:46.908 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:52.910 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:27:58.912 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:04.915 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:10.916 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:16.919 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:22.921 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:28.923 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:34.925 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:40.927 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:41.435 PM INFO org.apache.twill.yarn.YarnTwillController 
Failed to access application master.services application_1449175041089_0003 live node in ZK, resort to polling. Failure reason: KeeperErrorCode = NoNode for /instances/581f72d3-baac-4f1d-9a5a-4a8bc960c5e8
2:28:44.365 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Exception when fetching message on TopicPartition{topic=log, partition=0}.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67]
	at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67]
	at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_67]
	at kafka.network.BlockingChannel.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
	at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
2:28:44.407 PM WARN co.cask.cdap.internal.app.services.ProgramLifecycleService$RunRecordsCorrectorRunnable 
Unable to complete correcting run records: Service 'DatasetService' is not available. Please wait till it is up and running.
2:28:44.452 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0003 completed. Shutting down controller.
2:28:44.456 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Requesting stop of all consumer threads.
2:28:44.457 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Terminate requested Kafka-Consumer-log-0
2:28:44.459 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Wait for all consumer threads to stop.
2:28:44.459 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
All consumer threads stopped.
2:28:44.461 PM INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService 
Stopping KafkaClientService
2:28:44.461 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Stopping Kafka consumer
2:28:44.461 PM INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer 
Kafka Consumer stopped
2:28:44.462 PM INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService 
KafkaClientService stopped
2:28:44.462 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
master.services was terminated; restarting with back-off
2:28:46.929 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:50.583 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of dataset.executor Service to 1
2:28:50.589 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of metrics Service to 1
2:28:50.595 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of transaction Service to 1
2:28:50.600 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of streams Service to 1
2:28:50.605 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of explore.service Service to 1
2:28:50.614 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of log.saver Service to 1
2:28:50.626 PM INFO co.cask.cdap.data.runtime.main.MasterServiceMain 
Setting instance count of metrics.processor Service to 1
2:28:50.628 PM INFO co.cask.cdap.data.runtime.main.MasterTwillApplication 
Adding explore runnable.
2:28:50.633 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/core-site.xml
2:28:50.634 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/hdfs-site.xml
2:28:50.649 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/hive-site.xml
2:28:50.658 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/mapred-site.xml
2:28:50.659 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/ssl-client.xml
2:28:50.670 PM WARN co.cask.cdap.data.runtime.main.MasterServiceMain 
Ignoring duplicate config file: /usr/lib/hive/conf/yarn-site.xml
2:28:52.931 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:28:58.958 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:29:04.960 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:29:10.962 PM INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore 
RAMJobStore initialized.
2:29:13.605 PM INFO org.apache.twill.yarn.YarnTwillController 
Yarn application master.services application_1449175041089_0004 is in state RUNNING
2:29:16.964 PM INFO

kirakane nix

unread,
Dec 3, 2015, 5:41:32 PM12/3/15
to CDAP User, kira...@gmail.com

2015-12-03 14:22:37,239 INFO org.apache.twill.yarn.YarnTwillController: Failed to access application master.services application_1449175041089_0001 live node in ZK, resort to polling. Failure reason: KeeperErrorCode = NoNode for /instances/b0f464ec-20ff-42f3-b628-9f119ec2cdd4
2015-12-03 14:22:37,800 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:22:40,182 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Exception when fetching message on TopicPartition{topic=log, partition=0}.

java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67]
    at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_67]
    at kafka.network.BlockingChannel.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
    at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
2015-12-03 14:22:40,262 INFO org.apache.twill.yarn.YarnTwillController: Yarn application master.services application_1449175041089_0001 completed. Shutting down controller.
2015-12-03 14:22:40,270 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Requesting stop of all consumer threads.
2015-12-03 14:22:40,271 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Terminate requested Kafka-Consumer-log-0
2015-12-03 14:22:40,272 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Wait for all consumer threads to stop.
2015-12-03 14:22:40,273 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: All consumer threads stopped.
2015-12-03 14:22:40,274 INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService: Stopping KafkaClientService
2015-12-03 14:22:40,275 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Stopping Kafka consumer
2015-12-03 14:22:40,276 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Kafka Consumer stopped
2015-12-03 14:22:40,279 INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService: KafkaClientService stopped
2015-12-03 14:22:40,283 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: master.services was terminated; restarting with back-off
2015-12-03 14:22:42,403 WARN co.cask.cdap.internal.app.services.ProgramLifecycleService$RunRecordsCorrectorRunnable: Unable to complete correcting run records: Service 'DatasetService' is not available. Please wait till it is up and running.
2015-12-03 14:22:43,423 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of dataset.executor Service to 1
2015-12-03 14:22:43,430 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of metrics Service to 1
2015-12-03 14:22:43,436 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of transaction Service to 1
2015-12-03 14:22:43,441 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of streams Service to 1
2015-12-03 14:22:43,446 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of explore.service Service to 1
2015-12-03 14:22:43,451 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of log.saver Service to 1
2015-12-03 14:22:43,459 INFO co.cask.cdap.data.runtime.main.MasterServiceMain: Setting instance count of metrics.processor Service to 1
2015-12-03 14:22:43,459 INFO co.cask.cdap.data.runtime.main.MasterTwillApplication: Adding explore runnable.
2015-12-03 14:22:43,463 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: Ignoring duplicate config file: /usr/lib/hive/conf/core-site.xml
2015-12-03 14:22:43,464 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: Ignoring duplicate config file: /usr/lib/hive/conf/hdfs-site.xml
2015-12-03 14:22:43,476 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: Ignoring duplicate config file: /usr/lib/hive/conf/hive-site.xml
2015-12-03 14:22:43,484 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: Ignoring duplicate config file: /usr/lib/hive/conf/mapred-site.xml
2015-12-03 14:22:43,485 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: Ignoring duplicate config file: /usr/lib/hive/conf/ssl-client.xml
2015-12-03 14:22:43,492 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: Ignoring duplicate config file: /usr/lib/hive/conf/yarn-site.xml
2015-12-03 14:22:43,801 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:22:49,803 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:22:55,804 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:01,806 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:07,162 INFO org.apache.twill.yarn.YarnTwillController: Yarn application master.services application_1449175041089_0002 is in state RUNNING
2015-12-03 14:23:07,808 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:13,810 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:19,811 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:25,814 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:31,815 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:37,817 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:43,818 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:49,820 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:23:55,822 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:01,828 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:07,830 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:13,831 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:19,832 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:25,835 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:31,836 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:37,838 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:43,841 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:49,844 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:24:55,846 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:01,848 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:07,851 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:13,854 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:19,855 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:25,858 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:26,859 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:27,860 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:28,862 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:34,863 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:39,773 INFO org.apache.twill.yarn.YarnTwillController: Failed to access application master.services application_1449175041089_0002 live node in ZK, resort to polling. Failure reason: KeeperErrorCode = NoNode for /instances/587f7872-928a-4798-adfb-9ae6cd5c601f
2015-12-03 14:25:40,865 INFO co.cask.cdap.internal.app.runtime.schedule.store.DatasetBasedTimeScheduleStore: RAMJobStore initialized.
2015-12-03 14:25:42,681 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Exception when fetching message on TopicPartition{topic=log, partition=0}.

java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67]
    at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_67]
    at kafka.network.BlockingChannel.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.connect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.metrics.KafkaTimer.time(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) ~[org.apache.kafka.kafka_2.10-0.8.0.jar:0.8.0]
    at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
    at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) ~[org.apache.twill.twill-core-0.6.0-incubating.jar:0.6.0-incubating]
2015-12-03 14:25:42,791 INFO org.apache.twill.yarn.YarnTwillController: Yarn application master.services application_1449175041089_0002 completed. Shutting down controller.
2015-12-03 14:25:42,794 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Requesting stop of all consumer threads.
2015-12-03 14:25:42,795 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Terminate requested Kafka-Consumer-log-0
2015-12-03 14:25:42,796 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Wait for all consumer threads to stop.
2015-12-03 14:25:42,797 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: All consumer threads stopped.
2015-12-03 14:25:42,799 INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService: Stopping KafkaClientService
2015-12-03 14:25:42,800 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Stopping Kafka consumer
2015-12-03 14:25:42,801 INFO org.apache.twill.internal.kafka.client.SimpleKafkaConsumer: Kafka Consumer stopped
2015-12-03 14:25:42,802 INFO org.apache.twill.internal.kafka.client.ZKKafkaClientService: KafkaClientService stopped
2015-12-03 14:25:42,803 WARN co.cask.cdap.data.runtime.main.MasterServiceMain: master.services was terminated; restarting with back-off
20

On Thursday, December 3, 2015 at 1:49:42 PM UTC-8, kira...@gmail.com wrote:

Rohit Sinha

unread,
Dec 3, 2015, 6:21:52 PM12/3/15
to CDAP User, kira...@gmail.com
Hello Lawrence,
Can you make sure that cdap-kafka-server is running ?
You can run this command to get the status: 
/etc/init.d/cdap-kafka-server status

If its not running then please make sure that the property 'kafka.seed.brokers' in your cdap-site.xml found at /etc/cdap/conf/ is pointing to the correct path.
After this you can use the following command to restart all the cdap-services:
for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i restart ; done

Thanks, 
Rohit

kirakane nix

unread,
Dec 3, 2015, 7:11:00 PM12/3/15
to CDAP User, kira...@gmail.com
There is no init.d service setup for kafka.  I'm guessing this is not part of the the Parsel based install in CDH as far as I can tell.  I can see kafka running internally as part of the CDAP service in CDH. 

>ps aux |grep kafka

cdap     31188  0.7  1.2 3693320 110964 ?      Sl   14:17   0:43 /usr/java/jdk1.7.0_67-cloudera/bin/java -Dcdap.service=kafka-server -Xmx1073741824 -Dexplore.conf.files=/usr/lib/hive/conf/__cloudera_generation__:/usr/lib/hive/conf/__cloudera_

And I can see it running in the management console for CDH.

There is likewise just an empty cdap-site.xml in /etc/cdap/conf since I assume this is configured as part of the management console configuration in CDH.  I looked through the list of CDAP configuration properties in CDH and didn't find "kafka.seed.brokers".  

<configuration>
  <!--
    Your site level configuration goes here
  -->
</configuration>

Derek Wood

unread,
Dec 3, 2015, 8:53:45 PM12/3/15
to kirakane nix, CDAP User
Hi Lawrence,
Are you trying to run on a single VM?  It is very likely that you will not have enough capacity to run CM, Hadoop, and CDAP in a single VM.  When CDAP starts up, it launches a number of containers within Yarn, which combined will require ~14 vcores and ~14 Gb of memory (see http://docs.cdap.io/cdap/current/en/admin-manual/installation/hadoop/installation.html#hadoop-install-hardware-memory-core-requirements).  If the CDAP system containers cannot startup, nothing else will work.  

The first thing to check is your used/avail capacity from the Yarn ResourceManager UI.  And do you see the "cdap.master" container running?

Also note that for local CDAP development, we provide an SDK which gives you a CDAP installation without requiring Hadoop.  This can be downloaded from http://cask.co/downloads/

Thanks,
-Derek

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/f3c55b2c-cd9f-434e-afc1-1f2f5cd052a3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages