Hue Exception

[Greg Senia]

Hue Exception with Job Browser and HA-ResourceManager w/ YARN. Anyone ever seen this?

Traceback (most recent call last):

  File "/u01/hue_bin/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/handlers/base.py", line 112, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/u01/hue_bin/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/transaction.py", line 371, in inner

    return func(*args, **kwargs)

  File "/u01/hue_bin/apps/jobbrowser/src/jobbrowser/views.py", line 121, in jobs

    raise ex

RestException: Unable to authenticate <Response [200]>

<WSGIRequest

path:/jobbrowser/,

GET:<QueryDict: {u'state': [u'running'], u'user': [u'greg'], u'format': [u'json']}>,

POST:<QueryDict: {}>,

[Romain Rigaux]

This one is weird, 200 usually means ok: https://github.com/requests/requests-kerberos/blob/master/requests_kerberos/kerberos_.py

Does it work without HA? (cf other jira, there is either a bug with HA kerberos in Hue or a misconfiguration in YARN) Does it work when you curl the YARN REST API directly?

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[greg....@gmail.com]

Works fine without HA as we have 3 other clusters without HA and this is not logged all clusters configured the same other then auto failover with ResourceManager HA... I'd assume maybe this has something to do with the redirect that occurs under the covers when hitting the resource manager in stand by mode?

[Erick Tryzelaar]

Hello Greg,

Also, are you using Kerberos?

Thanks,

Erick

[greg....@gmail.com]

Yep using kerberos and its working for everything else correctly. Just not with Hue with HA ResourceManager

[Romain Rigaux]

I forgot (too many threads) but are you on latest Hue + which Hadoop version?

There was some cleanup in the HA retry code https://github.com/cloudera/hue/pull/225

and YARN HA redirect was improved

I just tried with our latest and it works:

[26/Aug/2015 10:43:51 -0700] resource     DEBUG    GET Got response: {"apps":{"app":[{"id":"applicati...

[26/Aug/2015 10:43:51 -0700] kerberos_    DEBUG    handle_response(): returning <Response [200]>

[26/Aug/2015 10:43:51 -0700] kerberos_    DEBUG    handle_mutual_auth(): Handling: 200

[26/Aug/2015 10:43:51 -0700] connectionpool DEBUG    "GET /ws/v1/cluster/apps?limit=10000&user=hive&finalStatus=UNDEFINED HTTP/1.1" 200 None

[26/Aug/2015 10:43:51 -0700] connectionpool INFO     Starting new HTTPS connection (1): nightly-kerberized-3.vpc.cloudera.com

[26/Aug/2015 10:43:51 -0700] cluster      WARNING  Picking RM HA: default

[26/Aug/2015 10:43:51 -0700] resource     DEBUG    GET Got response: {"clusterInfo":{"id":14406105583...

[26/Aug/2015 10:43:51 -0700] kerberos_    DEBUG    handle_response(): returning <Response [200]>

[26/Aug/2015 10:43:51 -0700] kerberos_    DEBUG    handle_mutual_auth(): Handling: 200

[26/Aug/2015 10:43:51 -0700] connectionpool DEBUG    "GET /ws/v1/cluster HTTP/1.1" 200 None

after stopping RM on kerb3

[26/Aug/2015 10:47:59 -0700] resource     DEBUG    GET Got response: {"apps":null}

[26/Aug/2015 10:47:59 -0700] kerberos_    DEBUG    handle_response(): returning <Response [200]>

[26/Aug/2015 10:47:59 -0700] kerberos_    DEBUG    handle_mutual_auth(): Handling: 200

[26/Aug/2015 10:47:59 -0700] connectionpool DEBUG    "GET /ws/v1/cluster/apps?limit=10000&user=hive&finalStatus=UNDEFINED HTTP/1.1" 200 None

[26/Aug/2015 10:47:59 -0700] connectionpool INFO     Starting new HTTPS connection (1): nightly-kerberized-1.vpc.cloudera.com

[26/Aug/2015 10:47:59 -0700] cluster      WARNING  Picking RM HA: ha

[26/Aug/2015 10:47:59 -0700] resource     DEBUG    GET Got response: {"clusterInfo":{"id":14406112160...

[26/Aug/2015 10:47:59 -0700] kerberos_    DEBUG    handle_response(): returning <Response [200]>

[26/Aug/2015 10:47:59 -0700] kerberos_    DEBUG    handle_mutual_auth(): Handling: 200

[26/Aug/2015 10:47:59 -0700] connectionpool DEBUG    "GET /ws/v1/cluster HTTP/1.1" 200 None

[26/Aug/2015 10:47:59 -0700] connectionpool INFO     Starting new HTTPS connection (1): nightly-kerberized-1.vpc.cloudera.com

[26/Aug/2015 10:47:59 -0700] cluster      ERROR    RM default is not available, skipping it: ('Connection aborted.', error(111, 'Connection refused'))
Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.1007/lib/hue/desktop/libs/hadoop/src/hadoop/cluster.py", line 191, in get_next_ha_yarncluster
    cluster_info = rm.cluster()
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.1007/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 86, in cluster
    return self._execute(self._root.get, 'cluster', params=kwargs, headers={'Accept': _JSON_CONTENT_TYPE})
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.1007/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 98, in _execute
    response = function(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.1007/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 97, in get
    return self.invoke("GET", relpath, params, headers=headers, allow_redirects=True)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.1007/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 78, in invoke
    urlencode=self._urlencode)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.1007/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 161, in execute
    raise self._exc_class(ex)
RestException: ('Connection aborted.', error(111, 'Connection refused'))

[26/Aug/2015 10:47:59 -0700] connectionpool INFO     Starting new HTTPS connection (1): nightly-kerberized-3.vpc.cloudera.com

[26/Aug/2015 10:47:59 -0700] cluster      WARNING  Picking RM HA: ha

It works too, the caching of the active is not done though: https://issues.cloudera.org/browse/HUE-2941

[greg....@gmail.com]

Hue 3.9 and Hadoop 2.6.0 plus a handful of fixes patched into HDP 2.2.6.0 code base

[greg....@gmail.com]

By changing: desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py   - All issues are solved .. Hue 3.9 now displays job data consistently and correct and no longer throws any errors.

+ return self._execute(self._root.get, 'cluster/info', params=kwargs, headers={'Accept': _JSON_CONTENT_TYPE})

On Wednesday, August 26, 2015 at 1:55:13 PM UTC-4, Romain Rigaux wrote:

I forgot (too many threads) but are you on latest Hue + which Hadoop version?

By changing the YARN code base

[Romain Rigaux]

Ha finally, glad to here!