CASK Master Service Won't Start in Cloudera Manager

192 views
Skip to first unread message

Dustin Davidson

unread,
Mar 22, 2017, 9:02:39 PM3/22/17
to CDAP User
Test driving CASK, and I installed via a Cloudera Parcel, but the Master Service starts then stops.  I the UI service is also failing so I assume the Master Service is causing this.  I am running CDH 5.10 and Cask version 4.1.0-2.  The Master Service logs are complaining about not connecting to Zookeeper.  Below is the failure message.  All other CASK services are fine and Zookeeper is up and available.


Mar 22, 7:41:56.429 PMERRORco.cask.cdap.common.runtime.DaemonMain
Exception when shutting down: Not connected to zooKeeper.
java.lang.IllegalArgumentException: Not connected to zooKeeper.
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) ~[com.google.guava.guava-13.0.1.jar:na]
	at org.apache.twill.internal.zookeeper.DefaultZKClientService.getZooKeeper(DefaultZKClientService.java:344) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.internal.zookeeper.DefaultZKClientService.doCreate(DefaultZKClientService.java:147) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.internal.zookeeper.DefaultZKClientService.create(DefaultZKClientService.java:137) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.zookeeper.ForwardingZKClient.create(ForwardingZKClient.java:61) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.internal.zookeeper.FailureRetryZKClient.create(FailureRetryZKClient.java:66) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.zookeeper.ForwardingZKClient.create(ForwardingZKClient.java:61) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.zookeeper.ZKClientServices$1.create(ZKClientServices.java:84) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.zookeeper.AbstractZKClient.create(AbstractZKClient.java:42) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.internal.zookeeper.ReentrantDistributedLock.acquire(ReentrantDistributedLock.java:233) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.internal.zookeeper.ReentrantDistributedLock.acquire(ReentrantDistributedLock.java:176) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at org.apache.twill.internal.zookeeper.ReentrantDistributedLock.lock(ReentrantDistributedLock.java:81) ~[org.apache.twill.twill-zookeeper-0.10.0.jar:0.10.0]
	at co.cask.cdap.data.runtime.main.MasterServiceMain.stop(MasterServiceMain.java:307) ~[na:na]
	at co.cask.cdap.common.runtime.DaemonMain$1.run(DaemonMain.java:44) ~[na:na]

Sagar Kapare

unread,
Mar 22, 2017, 9:33:35 PM3/22/17
to CDAP User
Hi Dustin,

Thank you for trying out CDAP!

Exception that you mentioned is happening during master shut down. 
Can you please attach the entire log here? This will give us hint about why master is getting stopped in the first place.

Thanks and Regards,
Sagar

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/2359303c-395e-4a43-a543-8f5b11986e31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dustin Davidson

unread,
Mar 23, 2017, 11:59:10 AM3/23/17
to CDAP User
Sagar,

Attached are the Logs.  Looks like the Yarn confirmation action is failing.  I checked the Yarn configs and all looks fine.  I also looked at the CDAP Yarn suggestions and think I have fulfilled those.  Any thoughts?
Sagar

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
master-service.log

Sagar Kapare

unread,
Mar 23, 2017, 2:51:23 PM3/23/17
to CDAP User
Hi Dustin,

Can you please also attach logs from stdout and stderr?

Thanks and Regards,
Sagar

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

Dustin Davidson

unread,
Mar 23, 2017, 8:21:20 PM3/23/17
to CDAP User
Attached are the logs.
stderr.log
stdout.log

Poorna Chandra

unread,
Mar 24, 2017, 5:23:13 PM3/24/17
to cdap...@googlegroups.com
Hi Dustin,

I see the following exception in the attached logs (full stack trace at the end of the message) - 
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=cdap/xxxxxxxxx, scope=cdap_system, params=[namespace=cdap_system],action=ADMIN)

Looks like cdap user does not have sufficient privileges to create HBase namespace cdap_system. Please grant cdap user privilege to create namespaces in HBase and restart CDAP Master.

Let me know if you  have any questions.

Thanks,
Poorna.

Stack trace:
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at co.cask.cdap.data.runtime.main.MasterServiceMain.main(MasterServiceMain.java:205)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=cdap/xxxxxxxxxx, scope=cdap_system, params=[namespace=cdap_system],action=ADMIN)
    at org.apache.hadoop.hbase.security.access.AccessController.requireNamespacePermission(AccessController.java:602)
    at org.apache.hadoop.hbase.security.access.AccessController.preGetNamespaceDescriptor(AccessController.java:1383)
    at org.apache.hadoop.hbase.master.MasterCoprocessorHost$7.call(MasterCoprocessorHost.java:170)
    at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1142)
    at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preGetNamespaceDescriptor(MasterCoprocessorHost.java:166)
    at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:2554)
    at org.apache.hadoop.hbase.master.MasterRpcServices.getNamespaceDescriptor(MasterRpcServices.java:817)
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55732)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at co.cask.cdap.data.runtime.main.MasterServiceMain.createSystemHBaseNamespace(MasterServiceMain.java:482)
    at co.cask.cdap.data.runtime.main.MasterServiceMain.start(MasterServiceMain.java:263)
    at co.cask.cdap.common.runtime.DaemonMain.doMain(DaemonMain.java:58)



To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

arche...@gmail.com

unread,
Jan 20, 2019, 5:13:45 AM1/20/19
to CDAP User
 am running CDAP in CDH quickstart, I am not sure what you mean by granting permission, but I tried following:

Enable HBase Authorization Using Cloudera Manager

  1. Go to Clusters and select the HBase cluster.
  2. Select Configuration.
  3. Search for HBase Secure Authorization and select it.
  4. Search for HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml and enter the following into it to enable hbase.security.exec.permission.checks. Without this option, all users will continue to have access to execute endpoint coprocessors. This option is not enabled when you enable HBase Secure Authorization for backward compatibility.
    <property>
      <name>hbase.security.exec.permission.checks</name>
      <value>true</value>
    </property>

under root user (default in CDH 5.x docker image shell):
sudo -H -u hbase bash -c  "hbase shell"

Then I did:
hbase(main):002:0> grant 'cdap', 'RWCA'
0 row(s) in 0.4130 seconds

Now the remaining issue is that CDAP UI service is still failing, but I think it is different issue....

I will open ticket to document it well on CDAP installation guide.

arche...@gmail.com

unread,
Jan 20, 2019, 5:48:51 AM1/20/19
to CDAP User
Actually after restart, there is another issue:
PriviledgedActionException as:yarn (auth:SIMPLE) cause:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=yarn, scope=cdap_system, params=[namespace=cdap_system,table=cdap_system:app.meta,family=d],action=CREATE)
	at org.apache.hadoop.hbase.security.access.AccessController.requireNamespacePermission(AccessController.java:634)
	at org.apache.hadoop.hbase.security.access.AccessController.preCreateTable(AccessController.java:1003)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:221)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1151)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preCreateTable(MasterCoprocessorHost.java:217)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1578)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)

Going to dig more how to resolve. I granted all permissions to yarn user (which is not correct of course, but I just try to test CDAP in CDH docker image:
hbase> grant 'cdap', 'RWCA'

I think this should be well documented in CDAP docs...

arche...@gmail.com

unread,
Jan 20, 2019, 5:52:24 AM1/20/19
to CDAP User
And now after yarn granted access, CDAP won't stop (already stopping for about 5-10 minutes) and high CPU usage....


Execute command Gracefully stop this CDAP Master Service on role CDAP Master Service (quickstart)

arche...@gmail.com

unread,
Jan 20, 2019, 7:11:20 AM1/20/19
to CDAP User
After these few restarts HBase stopped operating :-), it was due WALs generated, I removed those by:
[root@quickstart /]# sudo -H -u hdfs bash -c  "hdfs dfs -rm -r /hbase/WALs/*"

I wasn't able to get Hbase up and running due to issues with Java Heap space (set by CDH to default values), so  I increased and now I am able to access CDAP UI for first time, and all services reports GREEN status, happy to go and test the platform :-)
Reply all
Reply to author
Forward
0 new messages