Hue with Kerberos and SSL does not recognize Resource Manager

[Will Dailey]

Roman and team,

REQUEST: I have a request that would be very helpful to a number of us. Could you post onto your blog some idealized hue.ini files and state the conditions they are configured for. It certainly would be useful to have a reference for full HA with Kerberos and full SSL. Thanks in advanced.

Versions: CentOS 6.7, Java 1.7,  CDH 5.7, Hue came with the yum install. 

Install: I have installed HA for HDFS and YARN, Kerberos is working for all services, I installed SSL for the cluster using a Java Keystore/Truststore - I am using a self-signed cert in the truststore. This works for Hue. I am currently running a 12 node cluster on AWS. All of this is manually installed. There is no Cloudera Manager.

Situation: HDFS, YARN and MapReduce are working well from the command line. Beeline and Oozie are running from the command line. Hue comes up and I can log in, all of the services work but the Job Browser. 

Error: Hue does not recognize the Resource Manager. It is just not communicating.

Diag: My working theory is everything should now be communicating in HTTPS. I made a mapping of all of the secure ports I used in the configuration files and applied them to hue.ini. I have now written over 20 different combinations of variables and read a number of tickets. This is now as clear as a brick wall.

Configurations: These are the current set of configs for Hadoop. Your corrections are welcomed.

[hadoop]

        # Configuration for HDFS NameNode

        # ------------------------------------------------------------------------

        [[hdfs_clusters]]

                # HA support by using HttpFs

                [[[default]]]

                        # Enter the filesystem uri

                        #fs_defaultfs=hdfs://localhost:8020

                        fs_defaultfs=hdfs://mycluser

                        hdfs_port=8020

                        http_port=5007

                        # NameNode logical name.

                        logical_name=hdfs-ha

                        # WKD Use WebHdfs/HttpFs as the communication mechanism.

                        # Domain should be the NameNode or HttpFs host.

                        ## webhdfs_url=http://master01.invald:50070/webhdfs/v1

                        # Default port is 14000 for HttpFs. Required for HA

                        webhdfs_url=http://client01.invalid:14000/webhdfs/v1

                        # WKD Change this if your HDFS cluster is Kerberos-secured

                        ## security_enabled=false

                        security_enabled=true

                        # WKD In secure mode (HTTPS), if SSL certificates from YARN Rest APIs

                        # have to be verified against certificate authority

                        ## ssl_cert_ca_verify=True

                        ssl_cert_ca_verify=False

                        # Directory of the Hadoop configuration

                        ## hadoop_conf_dir=$HADOOP_CONF_DIR when set or '/etc/hadoop/conf'

        # Configuration for YARN (MR2)

        # ------------------------------------------------------------------------

        [[yarn_clusters]]

                [[[default]]]

                        # WKD Enter the host on which you are running the ResourceManager

                        ## resourcemanager_host=localhost

                        resourcemanager_host=master03.invalid

                        # The port where the ResourceManager IPC listens on

                        resourcemanager_port=8032

                        # Whether to submit jobs to this cluster

                        submit_to=True

                        # Resource Manager logical name (required for HA)

                        logical_name=rm1

                        # WKD Change this if your YARN cluster is Kerberos-secured

                        ## security_enabled=false

                        security_enabled=true

                        # WKD URL of the ResourceManager API

                        # resourcemanager_api_url=http://master03.invalid:8088

                        resourcemanager_api_url=http://master03.invalid:8090

                        # WKD URL of the ProxyServer API

                        #proxy_api_url=http://master03.invalid:8088

                        #proxy_api_url=http://master03.invalid:8090

                        # WKD URL of the HistoryServer API

                        history_server_api_url=http://master03.invalid:19888

                        # WKD In secure mode (HTTPS), if SSL certificates from YARN Rest APIs

                        # have to be verified against certificate authority

                        ## ssl_cert_ca_verify=True

                        ssl_cert_ca_verify=False

                # HA support by specifying multiple clusters.

                # Redefine different properties there.

                # e.g.

                [[[ha]]]

                        # WKD Enter the host on which you are running the ResourceManager

                        ## resourcemanager_host=localhost

                        resourcemanager_host=master04.invalid

                        # WKD The port where the ResourceManager IPC listens on

                        resourcemanager_port=8032

                       # Un-comment to enable

                        submit_to=True

                        # WKD Resource Manager logical name (required for HA)

                        logical_name=rm2

                        # WKD Change this if your YARN cluster is Kerberos-secured

                        # security_enabled=false

                        security_enabled=true

                        # WKD URL of the ResourceManager API

                        # resourcemanager_api_url=http://master04.invalid:8088

                        resourcemanager_api_url=http://master04.invalid:8090

                        # WKD URL of the ProxyServer API

                        ## proxy_api_url=http://master04.invalid:8088

                        # WKD URL of the HistoryServer API

                        history_server_api_url=http://master03.invalid:19888

                        # WKD In secure mode (HTTPS), if SSL certificates from YARN Rest APIs

                        # have to be verified against certificate authority

                        ## ssl_cert_ca_verify=True

                        ssl_cert_ca_verify=False

Error messages from error.log

[27/May/2016 00:02:18 -0700] cluster      ERROR    RM default is not available, skipping it: ('Connection aborted.', error(111, 'Connection refused'))

Traceback (most recent call last):

  File "/usr/lib/hue/desktop/libs/hadoop/src/hadoop/cluster.py", line 223, in get_next_ha_yarncluster

    cluster_info = rm.cluster()

  File "/usr/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 118, in cluster

    return self._execute(self._root.get, 'cluster/info', params=params, headers={'Accept': _JSON_CONTENT_TYPE})

  File "/usr/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 160, in _execute

    response = function(*args, **kwargs)

  File "/usr/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 98, in get

    return self.invoke("GET", relpath, params, headers=headers, allow_redirects=True)

  File "/usr/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 79, in invoke

    urlencode=self._urlencode)

  File "/usr/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 163, in execute

    raise self._exc_class(ex)

RestException: ('Connection aborted.', error(111, 'Connection refused'))

[27/May/2016 00:02:18 -0700] cluster      ERROR    RM ha is not available, skipping it: ('Connection aborted.', error(111, 'Connection refused'))

"error.log" 674L, 65818C

[Will Dailey]

Roman,

       This is resolved. This falls into the category of CARBON ERROR or just plain DUH!!. The issue is completely obvious once you see it. The URL for the resource manager api must prefix with https not http. Eariler I changed the port number but overlooked the s. The file is now edited to include:

     resourcemanager_api_url=https://master03.invalid:8090

     resourcemanager_api_url=https://master04.invalid:8090