Hdfs write permission denied when starting the mater

[陈布]

dear,all:

   I have successfully installed CDAP on RPM using Yum according to the page step by step.

    And I have created 3 directories/files on the in HDFS shown as following.

   

[root@node1 conf]# su hdfs

[hdfs@node1 conf]$ hdfs dfs -ls /

Found 8 items

drwxrwxrwx   - yarn   hadoop          0 2016-11-24 19:33 /app-logs

drwxr-xr-x   - hdfs   hdfs            0 2016-11-24 18:45 /apps

drwxr-xr-x   - yarn   hdfs            0 2016-11-25 10:45 /cdap

drwxr-xr-x   - hdfs   hdfs            0 2016-11-24 16:50 /hdp

drwxr-xr-x   - mapred hdfs            0 2016-11-24 16:50 /mapred

drwxrwxrwx   - mapred hadoop          0 2016-11-24 16:50 /mr-history

drwxrwxrwx   - hdfs   hdfs            0 2016-11-25 15:42 /tmp

drwxr-xr-x   - hdfs   hdfs            0 2016-11-25 11:04 /user

[hdfs@node1 conf]$ hdfs dfs -ls /cdap

Found 1 items

drwxr-xr-x   - yarn hdfs          0 2016-11-25 10:45 /cdap/tx.snapshot

[hdfs@node1 conf]$ hdfs dfs -ls /user/

Found 6 items

drwxrwx---   - ambari-qa hdfs          0 2016-11-24 18:46 /user/ambari-qa

drwxr-xr-x   - hcat      hdfs          0 2016-11-24 18:45 /user/hcat

drwx------   - hdfs      hdfs          0 2016-11-25 10:18 /user/hdfs

drwx------   - hive      hdfs          0 2016-11-24 18:45 /user/hive

drwxrwxr-x   - spark     hdfs          0 2016-11-24 16:50 /user/spark

drwxr-xr-x   - yarn      yarn          0 2016-11-25 14:03 /user/yarn

[hdfs@node1 conf]$ hdfs dfs -ls /user/yarn

Found 1 items

drwxr-xr-x   - yarn yarn          0 2016-11-25 14:03 /user/yarn/.hiveJars

   My cdap-site.xml (the path is /etc/cdap/conf/cdap-site.xml) is shown as following.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Copyright © 2014-2016 Cask Data, Inc.


  Licensed under the Apache License, Version 2.0 (the "License"); you may not
  use this file except in compliance with the License. You may obtain a copy of
  the License at


  http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
  License for the specific language governing permissions and limitations under
  the License.
  -->
<configuration>


  <!-- General Configuration -->


  <property>
    <name>hdfs.namespace</name>
    <value>/${root.namespace}</value>
    <description>
      Root directory for HDFS files written by CDAP
    </description>
  </property>
 
  <property>
    <name>hdfs.user</name>
    <value>yarn</value>
    <description>
      User name for accessing HDFS
    </description>
  </property>


  <property>
    <name>root.namespace</name>
    <value>cdap</value>
    <description>
      Root for this CDAP instance; used as the parent (or root) node for
      ZooKeeper, as the directory under which all CDAP data and metadata is
      stored in HDFS, and as the prefix for all HBase tables created by
      CDAP; must be composed of alphanumeric characters
    </description>
  </property>


  <property>
    <name>zookeeper.quorum</name>
    <value>10.32.180.40:2181,10.32.180.43:2181,10.32.180.46:2181/${root.namespace}</value>
    <description>
      ZooKeeper quorum string; specifies the ZooKeeper host:port; substitute the quorum
      (FQDN1:2181,FQDN2:2181,...) for the components shown here
    </description>
  </property>




  <!-- Applications Configuration -->


  <property>
    <name>app.bind.address</name>
    <value>0.0.0.0</value>
    <description>
      App Fabric service bind address
    </description>
  </property>




  <!-- Datasets Configuration -->


  <property>
    <name>data.tx.bind.address</name>
    <value>0.0.0.0</value>
    <description>
      Transaction service bind address
    </description>
  </property>




  <!-- Kafka Server Configuration -->


  <property>
    <name>kafka.default.replication.factor</name>
    <value>1</value>
    <description>
      CDAP Kafka replication factor; used to replicate Kafka messages across
      multiple machines to prevent data loss in the event of a hardware
      failure. The recommended setting is to run at least two CDAP Kafka servers.
      If you are running two Kafka servers, set this value to 2; otherwise,
      set it to the number of Kafka servers.
    </description>
  </property>
 
  <property>
    <name>kafka.log.dir</name>
    <value>/tmp/kafka-logs</value>
    <description>
      CDAP Kafka service log storage directory
    </description>
  </property>


<!--   <property>
    <name>kafka.seed.brokers</name>
    <value>FQDN1:9092,FQDN2:9092</value>
    <description>
      Comma-separated list of CDAP Kafka service brokers; for distributed CDAP,
      replace with list of FQDN:port brokers
    </description>
  </property> -->




  <!-- Metrics Configuration -->
 
  <property>
    <name>metrics.query.bind.address</name>
    <value>0.0.0.0</value>
    <description>
      Metrics Query service bind address
    </description>
  </property>




  <!-- Router Configuration -->


  <property>
    <name>router.bind.address</name>
    <value>0.0.0.0</value>
    <description>
      CDAP Router service bind address
    </description>
  </property>
 
  <property>
    <name>router.bind.port</name>
    <value>11111</value>
    <description>
      CDAP Router service bind port
    </description>
  </property>


  <property>
    <name>router.server.address</name>
    <value>{ROUTER-HOST-IP}</value>
    <description>
      CDAP Router service address to which CDAP UI connects
    </description>
  </property>


  <property>
    <name>router.server.port</name>
    <value>${router.bind.port}</value>
    <description>
      CDAP Router service port to which CDAP UI connects
    </description>
  </property>




  <!-- UI Configuration -->
 
  <property>
    <name>dashboard.bind.port</name>
    <value>9999</value>
    <description>
      CDAP UI bind port
    </description>
  </property>


</configuration>

But I failed to start cdap-master and catching this exception.

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=cdap, access=WRITE, inode="/user/cdap":hdfs:hdfs:drwxr-xr-x

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)

at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771)

at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755)

at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738)

at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:507)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

I have used the default config which hdfs.user is yarn. Why would the cdap-master throw this exception that seems to use 'cdap' to access HDFS?

Any help will be appreciated!

[陈布]

I create the HDFS directory '/user/cdap', the master throws error 'Table 'TableId{namespace=cdap_system, tableName=configuration}' does not exist after waiting 5000 ms. Giving up.' THe whole master-log is shown in the attachment.

Need help!!

在 2016年11月25日星期五 UTC+8下午4:16:38，陈布写道：

[Terence Yim]

HI,

It seems like it is caused by HBase master not able to assign region server for the cdap_system:configuration table regions. Can you check the HBase master log to see if it shows any error why the table was created?

Terence

-- 
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/eee3a2bf-73b6-46c2-b1a0-e080eaf75dee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<master-cdap-node2.hde.h3c.com.log>

[陈布]

Hi,Terence:

    Thanks a lot first.You are right. I found this 'The table cdap_system:configuration does not exist in meta but has a znode.' in the HBase master log.

    After I clean up the dirty data in Zookeeper, the CDAP master start up. But the CDAP services are still not available.

   The whole master-log is in the attachment.

    Could you give some suggestion  to fix the new problem? 

在 2016年11月26日星期六 UTC+8上午12:16:02，Terence Yim写道：

[Terence Yim]

Hi,

I don’t see anything particularly wrong from the cdap master log. How do you check the availability of the CDAP? Are you going through the CDAP UI? Note that when CDAP starts, it takes time for it to launch all the necessary system services on YARN, and the process can take up to couple minutes.

Terence

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/89f6a0ad-b259-486f-8f28-a1929ad12cad%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

<master-cdap-node1.hde.h3c.com.log>

[陈布]

hi,Terence:

      I check the availability of the CDAP through the CDAP UI which says  'CDAP Services are not available,trying to connect'.

      It has been several days since I started the CDAP, And it still shows the same information like the picture.

      All services(UI service,router service,master service,kafka service and auth service) processes seems to run pretty well.

       I type the command 'ps -ef | grep cdap',all service processes show up. And I use command 'netstat -ant | grep LISTEN' to

       find out all necessary service  ports are listening..And I found no particularly error from all service logs.

       

       The wired thing is that there is no log found in the router log or ui log when I visit the CDAP UI.

      Have you some suggestions about how to check the CDAP services or how to find out the reason why CDAP UI shows all services are not availables?

在 2016年11月29日星期二 UTC+8上午1:35:05，Terence Yim写道：

[Ali Anwar]

Hi.

Based upon the logs, it seems like CDAP master has started up properly. To determine whether it is an issue strictly with the UI, can you try making a REST call to the namespaces API with curl or some other tool that can make HTTP requests.
Reference documentation. For example:
curl -v <host>:<port>/v3/namespaces

If this works successfully (returning 200), then likely the UI is misconfigured.

Regards,

Ali Anwar

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/8b922998-46a7-472b-b31f-cf6b16a31d2b%40googlegroups.com.

[陈布]

Hi,Ali,

     You are right,it's caused by UI misconfigure. I check the cdap-site.xml and rewrite the porperty named 'router.server.port' and 'router.server.address'.

     And I can visit the CDAP UI now. Thanks a lot.

, And I can

在 2016年12月6日星期二 UTC+8上午3:14:54，ali写道：

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.