Transportation mode : binary and http mode

[pk.4...@gmail.com]

Hi,

We are using default transportation mode of HiveServer2 (binary) and it spawns a new thread for each client connection, and each thread remains alive until the client connection is closed because it uses TThreadPoolServer but the proxy doesn’t tolerate such connection and closes it immediately. 

The other transportation mode of HS2 is http and it is proxy friendly protocol over TCP. We have not enabled it. We can run HS2 simultaneously in both binary and http transport modes.

So new JDBCC connection string would look like this :

jdbc:hive2://{host}[:{port}][/{database}];ssl=true;sslTrustStore=<path>/mr3-ssl.jks;sslTrustStorePassword=<password>;transportMode=http;httpPath=<http_path>

Can we:
1. enable both binary and http transport modes

2. Is there anything extra we need to do other than updating XML files? 

3. Any SSL specific changes?

Thanks,

- Praveen

[Sungwoo Park]

Hello,

1.

My understanding is that a client can choose its own transport mode, but cannot use both binary and http modes. (What should happen if a client tries to connect to HS2 using both binary and http modes?) So, a client should use the same transport mode used by the HS2 that it is trying to connect to. 

If you can run two HS2 instances (e.g., one with binary transport and the other with http transport), the client could connect to the compatible HS2 by using its local address. If the client cannot directly connect to HS2, then you could create a dedicated Service for each HS2 and expose the address of the Services to clients.

2.

Hive 4 implements HS2 supporting both binary and http transport modes simultaneously. See HIVE-5312 (https://issues.apache.org/jira/browse/HIVE-5312), which is already merged to Hive 4 on MR3 . In this case, a client can use any transport mode. For Hive 3, this is not implemented.

Cheers,

--- Sungwoo 

[pk.4...@gmail.com]

Thanks Sungwoo,
As you know, we've two HS2 - we will explore one of them using http mode.

1. How was your experience with Hive4 w.r.t. stability? 

2. Anything (missing or bugs) you noticed that would make us to wait before trying it? I. understand it's still in beta and should not be used in the production.

- Praveen.

[Sungwoo Park]

Hi Praveen,

There seems to be a problem with HTTP transport on Kubernetes, and I am looking into this issue. If you set hive.server2.transport.mode to http, HiveServer2 may crash and repeat creating itself. Let me get back to you when I make progress in fixing the problem.

In my opinion, Hive 4 is at least as stable as Hive 3 for common use cases. While Hive 3 is widely used in production, it still has a lot of known bugs, many of which are being addressed in Hive 4 and some of which are unlikely to be fixed in the near future. You could check the increasing number of JIRA tickets in the Apache Hive site.

However, some breaking change may be introduce in Hive 4, so there is a risk in trying Hive 4 in production. If you can quickly generate a Metastore database, I think the risk in trying Hive 4 is small.

Cheers,

--- Sungwoo

[Sungwoo Park]

It turns out that HTTP transport works okay. HiveServer2 kept restarting just because I forgot to update the liveness/readiness probes. Let me create a new MR3docs page on using HTTP transport, and get back to you.

If you would like to try now, here is an example of JDBC connection string. It uses both Kerberos and SSL, and you can ignore principal=hive/gold7@PL if you don't use Kerberos.

jdbc:hive2://orange1:10001/;principal=hive/gold7@PL;;ssl=true;sslTrustStore=/home/gitlab-runner/mr3-run/kubernetes/beeline-ssl.jks;trustStorePassword=beelinepasswd1;transportMode=http;httpPath=/cliservice

* The port number 10001 is specified in the configuration key hive.server2.thrift.http.port in hive-site.xml.

* You should update readinessProbe and livenessProbe in hive.yaml. Using tcpSocket with port 10001 works, and with SSL, you cannot use httpGet with scheme HTTPS (currently not supported by Kubernetes).

* You should update yaml/hiveserver2-service.yaml to use port 10001.

If you cannot run multiple HS2 instances, then applying HIVE-5312 (https://issues.apache.org/jira/browse/HIVE-5312) back to Hive 3 seems to be an option. I will try to apply the patch to Hive 3 some time.

Cheers,

--- Sungwoo

[pk.4...@gmail.com]

Thank you, Sungwoo.

Please see my reply inline.

On Friday, September 25, 2020 at 6:53:03 AM UTC-7 Sungwoo Park wrote:

It turns out that HTTP transport works okay. HiveServer2 kept restarting just because I forgot to update the liveness/readiness probes. Let me create a new MR3docs page on using HTTP transport, and get back to you.

This is great - minimum change to get it running.

 

If you would like to try now, here is an example of JDBC connection string. It uses both Kerberos and SSL, and you can ignore principal=hive/gold7@PL if you don't use Kerberos.

jdbc:hive2://orange1:10001/;principal=hive/gold7@PL;;ssl=true;sslTrustStore=/home/gitlab-runner/mr3-run/kubernetes/beeline-ssl.jks;trustStorePassword=beelinepasswd1;transportMode=http;httpPath=/cliservice

Here you're using beeline trustStore - just an example to show the connection string. Correct?

 

* The port number 10001 is specified in the configuration key hive.server2.thrift.http.port in hive-site.xml.

* You should update readinessProbe and livenessProbe in hive.yaml. Using tcpSocket with port 10001 works, and with SSL, you cannot use httpGet with scheme HTTPS (currently not supported by Kubernetes).

[2] Does it mean the readiness/liveness probes will fail always? I mean is there any workaround so that probes are working fine? 

* You should update yaml/hiveserver2-service.yaml to use port 10001.

If you cannot run multiple HS2 instances, then applying HIVE-5312 (https://issues.apache.org/jira/browse/HIVE-5312) back to Hive 3 seems to be an option. I will try to apply the patch to Hive 3 some time.

 

[4] It is very relieving - most of the users want to use http and a few users binary. It would be great, if dual modes can be integrated into `Hive 3`. We already have multiple HS2 in binary mode and don't prefer to have another one for various reasons (cost, maintenance etc). Any ETA on this change?

 

On a side note, what is timeout you're using for LoadBalancer to avoid frequent loss of connection? I think we are using default - which is 1 60 seconds.

[Sungwoo Park]

If you would like to try now, here is an example of JDBC connection string. It uses both Kerberos and SSL, and you can ignore principal=hive/gold7@PL if you don't use Kerberos.

jdbc:hive2://orange1:10001/;principal=hive/gold7@PL;;ssl=true;sslTrustStore=/home/gitlab-runner/mr3-run/kubernetes/beeline-ssl.jks;trustStorePassword=beelinepasswd1;transportMode=http;httpPath=/cliservice

Here you're using beeline trustStore - just an example to show the connection string. Correct?

Yes.

 

 

* The port number 10001 is specified in the configuration key hive.server2.thrift.http.port in hive-site.xml.

* You should update readinessProbe and livenessProbe in hive.yaml. Using tcpSocket with port 10001 works, and with SSL, you cannot use httpGet with scheme HTTPS (currently not supported by Kubernetes).

[2] Does it mean the readiness/liveness probes will fail always? I mean is there any workaround so that probes are working fine? 

 tcpSocket with port 10001 works fine. httpGet with scheme HTTPS fails.

 

* You should update yaml/hiveserver2-service.yaml to use port 10001.

If you cannot run multiple HS2 instances, then applying HIVE-5312 (https://issues.apache.org/jira/browse/HIVE-5312) back to Hive 3 seems to be an option. I will try to apply the patch to Hive 3 some time.

 

[4] It is very relieving - most of the users want to use http and a few users binary. It would be great, if dual modes can be integrated into `Hive 3`. We already have multiple HS2 in binary mode and don't prefer to have another one for various reasons (cost, maintenance etc). Any ETA on this change?

I only added it to the TODO list, and don't have an ETA. This patch has been applied to Hive 2 as well as Hive 4, I am just hoping that it is not very hard to apply it to Hive 3.

 

 

On a side note, what is timeout you're using for LoadBalancer to avoid frequent loss of connection? I think we are using default - which is 1 60 seconds.

On AWS, we usually set it to a large value like 1000 seconds, but this has not been much of a concern. I guess you could find a value appropriate for your users.

Cheers,

--- Sungwoo

[Praveen Kumar]

Hi Sungwoo,

Let me know the ETA for the patch in Hive3. Currently, I am configuring HiveMR3 to set the transport mode (and thrift port) using env.sh, hive-setup.sh.

Do I have to do anything extra for HTTPS ? I am thinking it should work fine as long as the port is open and allowed in the security group. 

Thanks,

- Praveen.

--
You received this message because you are subscribed to a topic in the Google Groups "MR3" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hive-mr3/Lh19PZjx-sM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/6d875a4e-f5a2-4ae5-93e3-8ae5d7461a87n%40googlegroups.com.

[Sungwoo Park]

Hi Praveen,

We will try to apply the patch to Hive 3 early next week. At a glance, it appears to be feasible, but the patch is not compatible with Hive on MR3, and it may take a few days (if it is feasible at all). 

I have not found anything extra for HTTPS. If SSL is used, HiveServer2 uses HTTPS. (We have tested hive.server2.transport.mode=http on Hadoop and K8s, using Hive 2/3/4, with and without SSL/Kerberos, and everything works fine.) For liveness/readiness probes when SSL is enabled, you can use something like this:

        - containerPort: 10001

          protocol: TCP

        readinessProbe:

          tcpSocket:

            port: 10001

          initialDelaySeconds: 30

          periodSeconds: 30

        livenessProbe:

          tcpSocket:

            port: 10001

          initialDelaySeconds: 30

          periodSeconds: 30

Cheers,

--- Sungwoo

[Sungwoo Park]

I have update the Docker image (1.2-SNAPSHOT) after applying HIVE-5312. Now a single HiveServer2 instance can serve both Thrift and HTTP modes. 

For usage, see a new page in MR3docs and changes in mr3-run-k8s:

https://mr3docs.datamonad.com/docs/k8s/advanced/transport/

https://github.com/mr3project/mr3-run-k8s

If you are using Beeline, you can set HIVE_SERVER2_JDBC_OPTS in env.sh to append a string to JDBC connection string. See for examples:

https://mr3docs.datamonad.com/docs/k8s/guide/run-beeline/

I have tested the new patch using Hadoop and K8s, and with/without Helm. If you have any problem, please let me know.

Cheers,

--- Sungwoo

[Praveen Kumar]

Thank you, Sungwoo.

I was about to click on the "Send" button to ask about "service" for http mode and voila I saw your email about supporting dual modes. Much appreciated!

I will give it a try today and let you know if I have any questions/problems.

Once again thanks a lot for such a quick response.

Best,

- Praveen.

You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/3e198ff2-5ded-4558-8627-3f1f21143d5fn%40googlegroups.com.

[Sungwoo Park]

Hi,

If you use SSL but should use Python to access HiveServer2, the HTTP transport mode cannot be used with impyla, which does not support Kerberos/SSL + HTTP.

Cheers,

--- Sungwoo

[pk.4...@gmail.com]

Few questions:

1. As you're aware we create two HS2 servers (one for external use and one for internal services). We would like to keep it simple for internal use (aka it will be using single mode `binary` transport mode). Will HiveMR3 still create two services for single mode HS2 also or only if mode is `all` ?
2. I am using java properties hive.server2.authentication.mode to set transport mode as seen below. For external HS2 it will be set to `all` otherwise `binary`. Do I need to pass this Java property to `hiveserver2-servive.sh` too? I noticed that you are passing httpPort to it (I was making mistake by passing to hiveserver2-setup.sh only).
   <property>
   <name>hive.server2.transport.mode</name>
   <value>${hive.server2.transport.mode}</value>
   </property>
   
3. 
   

Thanks.

[pk.4...@gmail.com]

typo (copy/paste) mistake in #2 should be hive.server2.transport.mode not hive.server2.authentication.mode .

[Sungwoo Park]

1. If the transport mode is set to binary, only one service for binary transport is created. Two services are created only if the transport mode is set to all.

2. Yes, if you would like to use ${hive.server2.transport.mode} in hive-site.xml, you should use -Dhive.server2.transport.mode. See hive_setup_server2_update_hadoop_opts() in hive/hive/hiveserver2-service.sh for an example.

Thanks,

--- Sungwoo

[pk.4...@gmail.com]

One more request : can we maintain two tags (latest and versioned) for snapshot too? At some point, we may lock-in the tag for sometime before changing to latest. For example, I wanted to go back to previous snapshot (before this change) to see if the issue, I am seeing is in LoadBalancer/dns/security groups or new image. JDBC client can not connect to HS2. It errors out "Could not open client transport with JDBC Uri:<JDBC URL> .. Error creating the transport". It could be because of two reasons:

1. I am using Java property to set the hive.server2.transport.mode 
   
2. I have added port 10001 into LB/dns/security group's custom TCP.

The above changes were made (before you made above change) to have one mode per customer.

I will provide more info soon. I am pulling the latest snapshot and trying again.

[pk.4...@gmail.com]

Thanks Saungoo,
I am making mistake of not passing hive.server2.transport.mode in hiveserver2-service.sh . Let me try it out. I will update soon. 

[pk.4...@gmail.com]

Master node was not coming up then I realized that I had customized `run-master/worker.sh`  for AWS credential provider and bucket name. Master node was up and running when I added transport mode and httpPort - I am not sure which one fixed the issue :-}

[Sungwoo Park]

If you updated run-master/worker.sh, does this mean that you rebuild the Docker iamge?

In case that you forgot, if you can rebuild the Docker image, there is a page in MR3docs on adding custom configuration variables.

https://mr3docs.datamonad.com/docs/k8s/advanced/custom-config/

Cheers,

--- Sungwoo

[Praveen Kumar]

Hi Sungwoo,

Yes, we rebuild the docker image based on HiveMR3 image. We have many customizations (most are in hive-setup.sh, few in master/worker and now in hiverser2-service.sh too after this change for transport mode).

All pods and services are up and running. Testing the http mode with SSL.

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/2675e10e-1c2c-4a4d-9ad7-c11aeebec8b5n%40googlegroups.com.

[pk.4...@gmail.com]

Hi Sungwoo,

binary mode connectivity is fine, whereas I am getting errors on http mode. 

------------------------------

An error occurred while establishing the connection:

Long Message:
Could not open client transport with JDBC Uri: jdbc:hive2://<hs2-host-name>:10001/default;ssl=true;sslTrustStore=/Users/pkumar/key/mr3-ssl.jks;transportMode=http;httpPath=/cliservice: Could not create an https connection to jdbc:hive2://<hs2-host-name>:10001/default;ssl=true;sslTrustStore=/Users/pkumar/key/mr3-ssl.jks;transportMode=http;httpPath=/cliservice. null

Details:
   Type: java.sql.SQLException
   SQL State: 08S01

--------------------------------

JDBC Connection URL:

1. Binary mode:
   
   jdbc:hive2://<hs2-host-name>:9852/default;ssl=true;sslTrustStore=/Users/pkumar/key/mr3-ssl.jks;
2. Http mode:
   jdbc:hive2://<hs2-host-name>:10001/default;ssl=true;sslTrustStore=/Users/pkumar/key/mr3-ssl.jks;transportMode=http;httpPath=/cliservice
   

fyi: 

1. I did not get any errors when I tested transport mode "all" in minikube. I was able to connect using both modes.
2. I do dot see anything in the HS2 log.

Please let me know how to debug http connection issues.

Thanks.

[Sungwoo Park]

For HTTP mode, could you try after adding  trustStorePassword=beelinepasswd1? E.g.: 

jdbc:hive2://orange1:10001/;principal=hive/gold7@PL;;ssl=true;sslTrustStore=/home/gitlab-runner/mr3-run/kubernetes/beeline-ssl.jks;trustStorePassword=beelinepasswd1;transportMode=http;httpPath=/cliservice 

For me, I would set the transport mode to all, for both the internal and external HS instances. Then, you would not have to introduce another field in Helm values.yaml, and you would not have to rebuild the Docker image. There is a small memory overhead in HiveServer2 for running both HTTP and binary services, but in practice, you can ignore the memory overhead.

Cheers,

-- Sungwoo

[pk.4...@gmail.com]

I had tried with and without password and still could not connect using 'http' mode. I tried again with the password - it woks in binary mode but not in http mode.

Just like that I used trustStorePassword=beelinepasswd1 and it threw " Keystore was tampered with, or password was incorrect" error.

Any other info I need to provide you for troubleshoot? I don't see any activity/log in HS2 logs when I connect to it.

[pk.4...@gmail.com]

When I searched for SQL State: 08S01 and http mode connection error,  a very old issue was also returned : https://issues.apache.org/jira/browse/HIVE-11720 
Somewhere it was suggested following as workaround : (To resolve this issue, Set the following properties for a bigger HTTP header size in HiveServer properties:
hive.server2.thrift.http.response.header.size to 32768)

I did not find any reference to such property anywhere in Hive. doc https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2 .

[Sungwoo Park]

I added the same Docker image with a new tag (mr3project/hive3:1.2-SNAPSHOT.01). If this Docker image works, please let us know.

Cheers,

--- Sungwoo

[Praveen Kumar]

Thanks Sungwoo,

The new tag did the trick. I am testing external connectivity.

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/37c2994b-2f85-46d8-9968-afa52860f0d0n%40googlegroups.com.

[Praveen Kumar]

Hi Sungwoo,

After enabling SSL, transport mode binary works as usual. http mode connection fails (log attached - log is from external beeline command outside pod). There is no activity or log in the external-HS2 server.

I am seeing the same error on JDBC client DBVis too.

LoadBalancer has TCP protocol for both ports 10001 and 9852. Security group also has the same protocols and both ports are open.

[Sungwoo Park]

Hi Praveen,

I would suggest the following.

1. Check the configuration file kubernetes/conf/hive-site.xml:

<property>

  <name>hive.server2.transport.mode</name>

  <value>all</value>

</property>

<property>

  <name>hive.server2.use.SSL</name>

  <value>true</value>

</property>

2. Check if HiveServer2 creates two services (for http 10001 and binary 9852), e.g.:

2020-10-01T03:23:55,817  INFO [main] server.AbstractConnector: Started ServerConnector@65ef9a0a{SSL,[ssl, http/1.1]}{0.0.0.0:10001}

2020-10-01T03:23:55,817  INFO [main] server.Server: Started @87112ms

2020-10-01T03:23:55,817  INFO [main] thrift.ThriftCLIService: Started ThriftHttpCLIService in https mode on port 10001 path=/cliservice/* with 5...500 worker threads

2020-10-01T03:23:55,948  INFO [main] thrift.ThriftCLIService: Starting ThriftBinaryCLIService on port 9852 with 5...500 worker threads

3. Check the connection string. Here is an example of the connection string, and log messages printed by HiveServer2.

Connecting to jdbc:hive2://orange1:10001/;;ssl=true;sslTrustStore=/home/gitlab-runner/mr3-run/kubernetes/beeline-ssl.jks;trustStorePassword=beelinepasswd1;transportMode=http;httpPath=/cliservice

2020-10-01T03:27:13,812  INFO [HiveServer2-HttpHandler-Pool: Thread-38] thrift.ThriftHttpServlet: Could not validate cookie sent, will try to generate a new cookie

2020-10-01T03:27:13,813  INFO [HiveServer2-HttpHandler-Pool: Thread-38] thrift.ThriftHttpServlet: Cookie added for clientUserName gitlab-runner

2020-10-01T03:27:13,813  INFO [HiveServer2-HttpHandler-Pool: Thread-38] thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10

4. If Beeline fails to connect to HiveServer2, I can think of two potential problems: 1) Beeline cannot even reach port 10001 of HiveServer2; or 2) there is some problem with configuring Beeline.

To see if Beeline actually reaches port 10001 of HiveServer2, you can use telnet. 

1) In this example, we intentionally use port 10000 which is not open in HiveServer2. As a result, the connection fails.

$ telnet orange1 10000

Trying 192.168.10.1...

telnet: Unable to connect to remote host: Connection refused

2) In this example, we use port 10001 and the connection is established.

$ telnet orange1 10001

Trying 192.168.10.1...

Connected to orange1.

Escape character is '^]'.

In this case, HiveServer2 does not print any log messages. However, if you set logging level to DEBUG in kubernetes/conf/hive-log4j2.properties (by replacing INFO with DEBUG), HiveServer2 prints a lot of messages, as shown below. In this way, we can confirm whether or not telnet or Beeline can reach port 10001 of HiveServer2.

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-28-acceptor-0@57bac3f0-ServerConnector@2548fc01{SSL,[ssl, http/1.1]}{0.0.0.0:10001}] io.ManagedSelector: Queued change org.eclipse.jetty.io.ManagedSelector$Accept@4ea3643c on org.eclipse.jetty.io.ManagedSelector@4e9ea32f id=0 keys=0 selected=0

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.ManagedSelector: Selector loop woken up from select, 0/0 selected

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.ManagedSelector: Running change org.eclipse.jetty.io.ManagedSelector$Accept@4ea3643c

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.ManagedSelector: Queued change org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@160b1731 on org.eclipse.jetty.io.ManagedSelector@4e9ea32f id=0 keys=1 selected=0

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] strategy.ExecuteProduceConsume: EPC Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@55ed3a7a produced org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@160b1731

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] strategy.ExecuteProduceConsume: EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@55ed3a7a dispatch

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] strategy.ExecuteProduceConsume: EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@55ed3a7a run org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@160b1731

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-37] strategy.ExecuteProduceConsume: EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@55ed3a7a run

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-37] strategy.ExecuteProduceConsume: EPC Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@55ed3a7a produce enter

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] ssl.SslContextFactory: Customize 5f35f06e[SSLEngine[hostname=10.32.0.1 port=61307] SSL_NULL_WITH_NULL_NULL]

2020-10-01T03:40:51,390 DEBUG [HiveServer2-HttpHandler-Pool: Thread-37] strategy.ExecuteProduceConsume: EPC Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@55ed3a7a producing

2020-10-01T03:40:51,391 DEBUG [HiveServer2-HttpHandler-Pool: Thread-37] io.ManagedSelector: Selector loop waiting on select

2020-10-01T03:40:51,391 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] server.HttpChannel: new HttpChannelOverHttp@23b399fe{r=0,c=false,a=IDLE,uri=null} -> DecryptedEndPoint@7f4533cc{/10.32.0.1:61307<->10001,CLOSED,in,out,-,-,0/1800000,null@0}->SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,CLOSED,in,out,-,-,1/1800000,null@0}{io=0/0,kio=0,kro=0},null,HttpChannelState@41ff3143{s=IDLE a=NOT_ASYNC i=true r=NONE/false w=false}

2020-10-01T03:40:51,391 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] server.HttpConnection: New HTTP Connection HttpConnection@7e3c2ceb[DecryptedEndPoint@7f4533cc{/10.32.0.1:61307<->10001,CLOSED,in,out,-,-,0/1800000,null@0}->SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,CLOSED,in,out,-,-,1/1800000,null@0}{io=0/0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator@2831bbb3{s=START},c=HttpChannelOverHttp@23b399fe{r=0,c=false,a=IDLE,uri=null}]

2020-10-01T03:40:51,392 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.AbstractEndPoint: onOpen SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,Open,in,out,-,-,1/1800000,SslConnection@51fdda6e}{io=0/0,kio=0,kro=0}

2020-10-01T03:40:51,392 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.IdleTimeout: SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,Open,in,out,-,-,2/1800000,SslConnection@51fdda6e}{io=0/0,kio=0,kro=0} idle timeout check, elapsed: 2 ms, remaining: 1799998 ms

2020-10-01T03:40:51,392 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.AbstractConnection: onOpen SslConnection@51fdda6e{NOT_HANDSHAKING,eio=-1/-1,di=-1} -> HttpConnection@7e3c2ceb[DecryptedEndPoint@7f4533cc{/10.32.0.1:61307<->10001,Open,in,out,-,-,1/1800000,HttpConnection@7e3c2ceb}->SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,Open,in,out,-,-,2/1800000,SslConnection@51fdda6e}{io=0/0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator@2831bbb3{s=START},c=HttpChannelOverHttp@23b399fe{r=0,c=false,a=IDLE,uri=null}]

2020-10-01T03:40:51,392 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.AbstractConnection: onOpen HttpConnection@7e3c2ceb[DecryptedEndPoint@7f4533cc{/10.32.0.1:61307<->10001,Open,in,out,-,-,1/1800000,HttpConnection@7e3c2ceb}->SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,Open,in,out,-,-,2/1800000,SslConnection@51fdda6e}{io=0/0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator@2831bbb3{s=START},c=HttpChannelOverHttp@23b399fe{r=0,c=false,a=IDLE,uri=null}]

2020-10-01T03:40:51,392 DEBUG [HiveServer2-HttpHandler-Pool: Thread-38] io.AbstractConnection: fillInterested HttpConnection@7e3c2ceb[DecryptedEndPoint@7f4533cc{/10.32.0.1:61307<->10001,Open,in,out,-,-,1/1800000,HttpConnection@7e3c2ceb}->SelectChannelEndPoint@3139018a{/10.32.0.1:61307<->10001,Open,in,out,-,-,2/1800000,SslConnection@51fdda6e}{io=0/0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator@2831bbb3{s=START},c=HttpChannelOverHttp@23b399fe{r=0,c=false,a=IDLE,uri=null}]

5. If telnet connects to HiveServer2, then there is no problem of reaching port 10001 of HiveServer2. In this case, we can conclude that it's a configuration problem with Beeline. For me, I cannot reproduce this error, so it's hard for me to find a fix.

Let me know how the suggestion works.

Cheers,

--- Sungwoo

[Praveen Kumar]

Thanks Sungwoo - that was very helpful.

Checked 1-4 : everything seems to be fine. Telenet also worked. JDBC client DBVis and Beeline both are returning the same error.

Have you used any other clients in http mode? Or do you know? 

fyi: I have used https://ibis-project.org/docs/generated/ibis.impala.api.connect.html?highlight=hive in python's test driver code for external HS2 (for transport mode binary + SSL). I don't see any option for http transport mode.

I am going to enable DEBUG level logging and test the telnet.

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/92932736-87a8-445d-a503-f8c775aed80an%40googlegroups.com.

[Praveen Kumar]

fyi: DEBUG log for http at port 10001 seems to be the same as your -- attached log.  

Holy cow... 

Good news: It's working now.

Bad news: we lost a couple of days.

The reason: (I was suspicious of the character in the truststore password). I changed the password (pretty-print using only alpha-numeric), and generated self-signed certificates. Tried again and it worked.

Debugging via Telnet helped a lot as it was confirmed that the packet was reaching HS2.

Thanks a lot, Sungwoo for quick response with a new image tag and checking the logs and everything else..

PS: All is well if it ends well.

[Sungwoo Park]

I still have a lingering question: why did the binary mode with SSL work with the previous password? From my own testing using Beeline, we observe: 1) the password is optional in the connection string; 2) if it is provided, it should be a correct password (i.e., providing a wrong password fails the connection to HiveServer2).

At any rate, it's great that you have finally fixed the connection problem!

Cheers,

--- Sungwoo

[pk.4...@gmail.com]

That's great observation.

For binary mode, I did not had to supply the password as it was optional. For unknown reason, that is not true for http mode - I have to provide password.

Indeed it's great relief.

Thanks a ton. Your help is much appreciated.

[pk.4...@gmail.com]

Hello Sungwoo,

Has anyone able to use port 443 for http mode transport (if using SSL mode - self signed cert)? I am seeing error "Caused by: java.net.SocketException: Permission denied" as seen in the attached log "hs2-http-mode-port-443-log.txt.zip"

To avoid opening a new port to access, some of our customers are requesting if it could be available on port 443. Is it possible? Let me know.

fyi: Presently we are running it in dual mode (binary : 5862 and transport: 10001).

Thanks,

- Praveen.

[pk.4...@gmail.com]

fyi kubectl get services looks like

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                        AGE

hiveserver2                    LoadBalancer   <ip.address>    <aws-ip-address>.us-east-1.elb.amazonaws.com   9852:30420/TCP,443:30421/TCP   30s

[Sungwoo Park]

Hi Praveen,

A simple workaround is to create a new Docker image where setcap is installed, and then to exploit mr3.k8s.pod.worker.init.container.command in mr3-site.xml.

Here are the steps.

1. install setcap in Dockerfile

RUN apt-get update && apt-get install -y procps && apt-get install libcap2-bin && rm -rf /var/lib/apt/lists/*

2. run setcap in an init container by updating mr3-site.xml

<property>

  <name>mr3.k8s.pod.worker.init.container.command</name>

  <value>setcap 'cap_net_bind_service=+ep' /usr/local/openjdk-8/bin/java</value>

</property>

If mr3.k8s.pod.worker.init.container.command already contains a command, append the new command after ';'. (See https://mr3docs.datamonad.com/docs/k8s/advanced/run-non-root/)

3. change port to 443 by updating env.sh, hive.yaml, and hiveserver2-service.yaml. E.g.)

env.sh:

HIVE_SERVER2_PORT=443

hive.yaml:        

      ports:

        - containerPort: 443

          protocol: TCP

        readinessProbe:

          tcpSocket:

            port: 443

          initialDelaySeconds: 30

          periodSeconds: 30

        livenessProbe:

          tcpSocket:

            port: 443

          initialDelaySeconds: 30

          periodSeconds: 30

hiveserver2-service.yaml:

 

  ports:

  - protocol: TCP

    port: 443

    targetPort: 443

Then, HiveServer2 opens at port 443, e.g.:

Connecting to jdbc:hive2://blue0:443/;;

Connected to: Apache Hive (version 3.1.3)

Driver: Hive JDBC (version 3.1.3)

Transaction isolation: TRANSACTION_REPEATABLE_READ

Beeline version 3.1.3 by Apache Hive

0: jdbc:hive2://blue0:443/> 

Hope this helps,

Cheers,

--- Sungwoo

[Sungwoo Park]

Oops, I tested on a cluster where HiveServer2 runs as root :-( Let me test on another cluster where HiveServer2 runs as a normal user.

Cheers,

--- Sungwoo

[Sungwoo Park]

I tried the standard approach of adding NET_BIND_SERVICE capability to HiveServer2 Pod. Unfortunately it turns out that there is a bug in Kubernetes that causes SecurityContext not to work as intended. This bug has been discussed in depth (for over three years) in the following thread:

https://github.com/kubernetes/kubernetes/issues/56374

So, I had to employ a trick instead. If you would like to use ports lower than 1024 while running HiveServer2 as a non-root user, you can exploit an init container that calls sysctl. Here are the steps.

1. Update yaml/hive.yaml so that it starts with an init container that executes 'sysctl net.ipv4.ip_unprivileged_port_start=0'. You don't have to rebuild Docker images.

yaml/hive.yaml:

spec:

  template:

    spec:

      initContainers:

      - name: init-command

        image: orange1:5000/hive3:latest

        args:

        - sysctl

        - net.ipv4.ip_unprivileged_port_start=0

        securityContext:

          privileged: true

          runAsUser: 0

2. Update HIVE_SERVER2_HTTP_PORT in env.sh. Also update containerPort in hive.yaml, and port and targetPort in hiveserver2-service.yaml.

env.sh:

HIVE_SERVER2_HTTP_PORT=444

hive.yaml:

        ports:

        - containerPort: 9852

          protocol: TCP

        - containerPort: 444

          protocol: TCP

hiveserver2-service.yaml:

  - protocol: TCP

    port: 444

    targetPort: 444

    name: http

3. Then, you can connect to HiveServer2 at port 444.

Connected to: Apache Hive (version 3.1.3)

Driver: Hive JDBC (version 3.1.3)

Transaction isolation: TRANSACTION_REPEATABLE_READ

Beeline version 3.1.3 by Apache Hive

0: jdbc:hive2://orange1:444/> 

FYI, if you would like to try the standard approach (which I don't think is working reliably as of now), add SecurityContext to hive.yaml:

      containers:

      - image: orange1:5000/hive3:latest

        securityContext:

          capabilities:

            add: 

            - NET_BIND_SERVICE

        command: ["/opt/mr3-run/hive/hiveserver2-service.sh"]

You will see that the HiveServer2 does not actually possess NET_BIND_SERVICE capability because when Kubernetes changes the user ID from 0 to a non-zero value, all capabilities sets are cleared. (In the example below, CapEff is all cleared.) It looks like this is the main cause of the weird behavior regarding SecurityContext.

hive@hivemr3-hiveserver2-cw8bv:/opt/mr3-run/hive$ cat /proc/23/status | grep Cap

CapInh: 00000000a80425fb

CapPrm: 0000000000000000

CapEff: 0000000000000000

CapBnd: 00000000a80425fb

CapAmb: 0000000000000000

Depending on the configuration of your Kubernetes cluster, you may also have to create PodSecurityPolicy, ClusterRole, and ClusterRoleBinding resources, e.g.:

policy.yaml:

apiVersion: policy/v1beta1

kind: PodSecurityPolicy

metadata:

  name: privileged

  annotations:

    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'

spec:

  privileged: true

  allowPrivilegeEscalation: true

  allowedCapabilities:

  - '*'

  volumes:

  - '*'

  hostNetwork: true

  hostPorts: 

  - min: 0

    max: 65535

  hostIPC: true

  hostPID: true

  runAsUser:

    rule: 'RunAsAny'

  seLinux:

    rule: 'RunAsAny'

  supplementalGroups:

    rule: 'RunAsAny'

  fsGroup:

    rule: 'RunAsAny'

clusterrole.yaml:

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: psp-privileged

rules:

- apiGroups:

  - extensions

  resources:

  - podsecuritypolicies

  resourceNames:

  - privileged

  verbs:

  - use

clusterrolebinding.yaml:

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: psp-privileged

subjects:

- kind: ServiceAccount

  namespace: hivemr3

  name: hive-service-account

roleRef:

  kind: ClusterRole

  name: psp-privileged

  apiGroup: rbac.authorization.k8s.io

Thanks for asking, and hope this helps,

Cheers,

--- Sungwoo 

[Praveen Kumar]

Thanks Sungwoo,

I tried and here is my changes in hive.yaml under spec:

# initContainers to allow privileged-port for a non-root user.
initContainers:
  - name: init-command
    image: {{ .Values.docker.image }}
    imagePullPolicy: {{ .Values.docker.imagePullPolicy }}

    args:
      - sysctl
      - net.ipv4.ip_unprivileged_port_start=0
    securityContext:
      privileged: true
      runAsUser: 0

The initContainer is crash looping.

NAME                            READY   STATUS       RESTARTS   AGE
hivemr3-hiveserver2-qfq7k       0/1     Init:Error   6          6m7s

I attached the "kubectl describe pod" info. I tried with and without image pull policy but initContainer was still crashing. I have hiveserver2-service.yaml and env.sh correctly. How do I debug this? Let me know if you need more information. Thanks.

Another question (related to the same issue): Since the service is of loadBalancer type can it not accept the inbound at 443 and connect to service running at non-privileged ports?

Thanks again for the quick response.

- Praveen.

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/96f47ce9-30a5-4159-bc36-b5f3e85ae67dn%40googlegroups.com.

[Sungwoo Park]

1. For debugging init containers, check out the console output, e.g.:

kubectl logs -n hivemr3-hiveserver2-z5858 -c init-command  

I am not aware of any other way of debugging init containers because the console output is the only feedback from the command executed by init containers.

2. Yes, you can open a non-privileged port and forward traffic from a privileged port. Here is an example.

env.sh:

HIVE_SERVER2_HTTP_PORT=10001

--> so, HiveServer2 opens HTTP port at 10001.

hiveserver2-service.yaml:

spec:

  type: LoadBalancer

  ports:

  - protocol: TCP

    port: 444

    targetPort: 10001

--> so, we forward traffic from port 444 to HiveServer2 port 10001.

Then, you can connect to HiveServer2 through port 444 from the outside. E.g. from my own experiment:

Connecting to jdbc:hive2://orange1:444/;principal=hive/orange1@PL;ssl=true;sslTrustStore=/home/gitlab-runner/mr3-run/kubernetes/beeline-ssl.jks;trustStorePassword=beelinepasswd1;transportMode=http;httpPath=/cliservice

Connected to: Apache Hive (version 3.1.3)

Driver: Hive JDBC (version 3.1.3)

Transaction isolation: TRANSACTION_REPEATABLE_READ

Beeline version 3.1.3 by Apache Hive

0: jdbc:hive2://orange1:444/> 

Cheers,

--- Sungwoo

[Praveen Kumar]

I tried #2 (LoadBalancer's inbound TCP port 443 aka https) but the connection request is not reaching the HS2 server. I can connect via port 9852 (binary code). I received following error message:

jdbc:hive2://<hs-server-host>:443/<my-db>;ssl=true;sslTrustStore=<path-topublic-cert>/mr3-ssl.jks;transportMode=http;httpPath=cliservice;trustStorePassword=<my-password>;: Could not establish connection to jdbc:hive2://<hs-server-host>:443/<my-db>;ssl=true;sslTrustStore=<path-topublic-cert>/mr3-ssl.jks;transportMode=http;httpPath=cliservice;trustStorePassword=<my-password>;: org.apache.hive.org.apache.http.conn.HttpHostConnectException: Connect to <hs-server-host>:443 [<hs-server-host>/54.166.108.59, <hs-server-host>/18.215.72.191, <hs-server-host>/34.232.155.180] failed: Operation timed out (Connection timed out)

I will try #1 tomorrow and let you know.

--
You received this message because you are subscribed to a topic in the Google Groups "MR3" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hive-mr3/Lh19PZjx-sM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hive-mr3+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/71585c25-6784-4dc7-823e-d07141998d49n%40googlegroups.com.

[Sungwoo Park]

Could you check if the request reaches LoadBalancer at port 443, for example, by checking the inbound rule of LoadBalancer? On AWS, you might have to check the inbound rule separately.

--- Sungwoo

[Praveen Kumar]

Thank you, Sungwoo.

The issue was with the security group for LB. Added port 443 to the allowed list of inbound connections and it worked.

Best,

- Praveen.

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/a59a0c80-7b99-4014-95c2-f711faf5f6f8n%40googlegroups.com.