Can't access filesystem in Hue 3.10

[Dale Bradman]

Hello,

I am having problems connecting to the HDFS filesystem. 

I am using kerberos backed by AD and have successfully synchronised users and groups.

On the About Hue page, I see 

hadoop.hdfs_clusters.default.webhdfs_url

Current value: http://mh0edge01:14000/webhdfs/v1/ Failed to access filesystem root

It also fails to create the home directory when I login to Hue.

When I click on the FileBrowser tab I see the following error:

Cannot access: /user/daleb. Note: you are a Hue admin but not a HDFS superuser, "hdfs" or part of HDFS supergroup, "hdfs".

Unable to authenticate <Response [401]>

On the Hue server, I am able to kinit using Hue and access the HDFS filesystem. Does the hue keytab have to be present on every node?

Is there anything else that I am missing?

Thanks,

Dale

[Meilong]

Make sure your namenode port is 14000. It's 50070 by default.

在 2016年6月22日星期三 UTC+8下午6:53:07，Dale Bradman写道：

[Tatsuo Kawasaki]

Hi Dale,

Please see below link:

http://stackoverflow.com/questions/35187085/hue-failed-to-access-filesystem-root

Thanks,

--

Tatsuo

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

--

--
Tatsuo Kawasaki
tat...@cloudera.com

[Dale Bradman]

Hi Tatsuo,

I have those parameters already configured.

I have a Hue user in Active Directory. In the command line, I can kinit as this hue user and access the webhdfs filesystem at port 14000 using the following command:

curl -i --negotiate -u : "http://edge01:14000/webhdfs/v1/user/?op=LISTSTATUS"  <-- this command WORKS as user Hue.

Therefore, I can assume that hadoop-HTTPfs is correctly configured. I have also double checked by "Kerberos" section of Hue.ini file and this information is correct:

    # Path to Hue's Kerberos keytab file

    hue_keytab=/etc/security/keytabs/hue.service.keytab

    # Kerberos principal name for Hue

    hue_principal=hue

    # Path to kinit

    kinit_path=/usr/bin/kinit

Here are some error messages that I can see in Hue:

[29/Jun/2016 08:37:21 -0700] views        ERROR    Could not create home directory.
Traceback (most recent call last):
  File "/opt/hue/desktop/core/src/desktop/auth/views.py", line 130, in dt_login
    ensure_home_directory(request.fs, user.username)
  File "/opt/hue/apps/useradmin/src/useradmin/views.py", line 662, in ensure_home_directory
    fs.do_as_user(username, fs.create_home_dir, home_dir)
  File "/opt/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 86, in do_as_user
    return fn(*args, **kwargs)
  File "/opt/hue/desktop/core/src/desktop/lib/fs/proxyfs.py", line 149, in create_home_dir
    self._get_fs(home_path).create_home_dir(home_path)
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/hadoopfs.py", line 243, in create_home_dir
    if not self.exists(home_path):
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 242, in exists
    return self._stats(path) is not None
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 230, in _stats
    raise ex
WebHdfsException: Unable to authenticate <Response [401]>

[29/Jun/2016 08:37:23 -0700] webhdfs      ERROR    Failed to determine superuser of WebHdfs at http://edge01.hadoop.local:14000/webhdfs/v1/: Unable to authenticate <Response [401]>
Traceback (most recent call last):
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 149, in superuser
    sb = self.stats('/')
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 236, in stats
    res = self._stats(path)
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 230, in _stats
    raise ex
WebHdfsException: Unable to authenticate <Response [401]>

[29/Jun/2016 08:37:23 -0700] kerberos_    ERROR    handle_mutual_auth(): Mutual authentication failed

[29/Jun/2016 08:37:23 -0700] kerberos_    ERROR    authenticate_server(): authGSSClientStep() failed:
Traceback (most recent call last):
  File "/opt/hue/build/env/local/lib/python2.7/site-packages/requests_kerberos-0.6.1-py2.7.egg/requests_kerberos/kerberos_.py", line 229, in authenticate_server
    _negotiate_value(response))
GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Bad format in credentials cache', -1765328185))

[29/Jun/2016 08:37:23 -0700] kerberos_    DEBUG    authenticate_server(): Authenticate header: 

[29/Jun/2016 08:37:23 -0700] kerberos_    DEBUG    handle_mutual_auth(): Authenticating the server

[29/Jun/2016 08:37:23 -0700] kerberos_    DEBUG    handle_mutual_auth(): Handling: 401

[29/Jun/2016 08:37:23 -0700] kerberos_    DEBUG    handle_response(): returning <Response [401]>

[29/Jun/2016 08:37:23 -0700] kerberos_    DEBUG    handle_401(): returning <Response [401]>

[29/Jun/2016 08:37:23 -0700] kerberos_    ERROR    generate_request_header(): authGSSClientStep() failed:
Traceback (most recent call last):
  File "/opt/hue/build/env/local/lib/python2.7/site-packages/requests_kerberos-0.6.1-py2.7.egg/requests_kerberos/kerberos_.py", line 114, in generate_request_header
    _negotiate_value(response))
GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Bad format in credentials cache', -1765328185))

Thanks,

Dale

[Romain Rigaux]

Do you have the certificate verification set to true?

ssl_cert_ca_verify=True

Maybe this one has a problem, could quick check with false maybe.

[Romain Rigaux]

Ha reading more, last time we saw this, kt_renewer in Hue was not set, not set to renewable. Could you retry the curl on the Hue machine without doing any kinit? (kt_renewer should do it for you)

[Dale Bradman]

My ini file showed:

## ssl_cert_ca_verify=True

So I uncommented that line (in HDFS section only) and tried it - this did not work.
I then changed it to False which also did not work.


In regards to kt_renewer...

-->

root@mh0edge01:~# ssh hue@localhost

hue@localhost's password:

hue@mh0edge01:~$ curl -i --negotiate -u : "http://mh0edge01:14000/webhdfs/v1/user/?op=LISTSTATUS"

HTTP/1.1 401 Unauthorized

Server: Apache-Coyote/1.1

WWW-Authenticate: Negotiate

Set-Cookie: hadoop.auth=; Path=/; HttpOnly

Content-Type: text/html;charset=utf-8

Content-Length: 997

Date: Thu, 30 Jun 2016 17:03:04 GMT

HTTP/1.1 200 OK

Server: Apache-Coyote/1.1

WWW-Authenticate: Negotiate ***&*(%^*&(*$%^&*

Set-Cookie: hadoop.auth="u=hue&p=h...@AD.LOCAL&t=kerberos-dt&e=ftuasygduohldas="; Path=/; HttpOnly

Content-Type: application/json

Transfer-Encoding: chunked

Date: Thu, 30 Jun 2016 17:03:17 GMT

{"FileStatuses":{"FileStatus":[{"pathSuffix":"ambari-qa","type":"DIRECTORY","length":0,"owner":"ambari-qa","group":"hdfs","permission":"770","accessTime":0,"modificationTime":1464798497502,"blockSize":0,"replication":0},{"pathSuffix":"automation","type":"DIRECTORY","length":0,"owner":"automation","group":"hdfs","permission":"755","accessTime":0,"modificationTime":1467303178887,"blockSize":0,"replication":0},

<--

When I ssh to Hue user this automatically grants a kerberos ticket.

Therefore to remove the k-ticket I have done the following:

-->

root@mh0edge01:~# ssh hue@localhost

hue@localhost's password:

hue@mh0edge01:~$ kdestroy

hue@mh0edge01:~$ curl -i --negotiate -u : "http://mh0edge01:14000/webhdfs/v1/user/?op=LISTSTATUS"

HTTP/1.1 401 Unauthorized

Server: Apache-Coyote/1.1

WWW-Authenticate: Negotiate

Set-Cookie: hadoop.auth=; Path=/; HttpOnly

Content-Type: text/html;charset=utf-8

Content-Length: 997

Date: Thu, 30 Jun 2016 17:08:22 GMT

<html><head><title>Apache Tomcat/6.0.44 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - Authentication required</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Authentication required</u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.44</h3></body></html>hue@mh0edge01:~$

<--

Where is the kt_renewer property set? I cannot find it in the hue.ini file.

Thanks Romain,

Dale.

[Dale Bradman]

Any further thoughts on what I can do here?

Here is my krb.conf file if it helps:

[libdefaults]

  renew_lifetime = 7d

  forwardable = true

  default_realm = {{realm}}

  ticket_lifetime = 24h

  dns_lookup_realm = false

  dns_lookup_kdc = false

  #default_tgs_enctypes = {{encryption_types}}

  #default_tkt_enctypes = {{encryption_types}}

{% if domains %}

[domain_realm]

{% for domain in domains.split(',') %}

  {{domain}} = {{realm}}

{% endfor %}

{% endif %}

[logging]

  default = FILE:/var/log/krb5kdc.log

  admin_server = FILE:/var/log/kadmind.log

  kdc = FILE:/var/log/krb5kdc.log

[realms]

  {{realm}} = {

    admin_server = {{admin_server_host|default(kdc_host, True)}}

    kdc = {{kdc_host}}

  }

{# Append additional realm declarations below #}

On Thursday, June 30, 2016 at 10:49:13 PM UTC+1, Dale Bradman wrote:

My ini file showed:

## ssl_cert_ca_verify=True

So I uncommented that line (in HDFS section only) and tried it - this did not work.
I then changed it to False which also did not work.


In regards to kt_renewer...

-->

root@mh0edge01:~# ssh hue@localhost

hue@localhost's password:

hue@mh0edge01:~$ curl -i --negotiate -u : "http://mh0edge01:14000/webhdfs/v1/user/?op=LISTSTATUS"

HTTP/1.1 401 Unauthorized

Server: Apache-Coyote/1.1

WWW-Authenticate: Negotiate

Set-Cookie: hadoop.auth=; Path=/; HttpOnly

Content-Type: text/html;charset=utf-8

Content-Length: 997

Date: Thu, 30 Jun 2016 17:03:04 GMT

HTTP/1.1 200 OK

Server: Apache-Coyote/1.1

WWW-Authenticate: Negotiate ***&*(%^*&(*$%^&*

Set-Cookie: hadoop.auth="u=hue&p=hue@AD.LOCAL&t=kerberos-dt&e=ftuasygduohldas="; Path=/; HttpOnly

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+unsubscribe@cloudera.org.

--

--
Tatsuo Kawasaki
tat...@cloudera.com

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+unsubscribe@cloudera.org.

[Dale Bradman]

Bump!

Still very much unable to connect to HDFS in Hue.

I can kinit using hue:

root@edge:~# kinit -kt /etc/security/keytabs/hue.service.keytab hue/edge.hadoop.private

root@edge:~# klist -k -t /etc/security/keytabs/hue.service.keytab
Keytab name: FILE:/etc/security/keytabs/hue.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   4 01/01/70 01:00:00 hue/edge.hadoop.private@AD.PRIVATE

  [[kerberos]]

    # Path to Hue's Kerberos keytab file
    hue_keytab=/etc/security/keytabs/hue.service.keytab
    # Kerberos principal name for Hue

    hue_principal=hue/edge.hadoop.private@AD.PRIVATE
    # Path to kinit
    kinit_path=/usr/bin/kinit

Really stuck here! Thanks.