Using Waggle Dance with Qubole

[Audrey Y]

Hi Waggle Dance User Group,

I am trying to use waggle dance with qubole, and I am following this

guide 

https://docs.qubole.com/en/latest/user-guide/engines/hive/custom-hive-metastore.html#configuring-thrift-metastore-s

to configure custom metastore (which is waggle dance in this case). Based on my understanding, the waggle dance server listens on port 48869, however, in this guide, it says "The port that is used for Hive Metastore Service in Qubole Hive is 10000". I know that 10000 is the default port of HS2, but aren't we supposed to provide the waggle dance server's port 48869 instead of HS2's port 10000 for qubole to connect to waggle dance? In short, my question is how to make qubole use waggle dance. Thank you in advance!

Thanks,

Audrey

[Patrick Duin]

Hi Audrey,

I am no qubole expert but I think you are correct that the docs mix up HS2 thrift endpoint and Hive metastore thrift endpoint.

This is the correct property though: hive.metastore.uris


The default port for the metastore thrift uri is 9083.

I would think you need to replace any reference to uris with 9083 port in qubole with the waggle-dance uri and port. Waggle Dance doesn't proxy HS2.


You can test this in Qubole as well by overriding in query:
set hive.metastore.uris=thrift://<waggle>:48869;


select * from ...;


Hope this helps,

 Patrick

Op do 27 mei 2021 om 19:10 schreef Audrey Y <yuan...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "Waggle Dance User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to waggle-dance-u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/waggle-dance-user/14e36176-1e22-4a06-856f-bf1a5b487c9dn%40googlegroups.com.

[Pradeep Bhadani]

Hi Audrey,

As Patrick suggested, you need to update the Hive Metastore property (not HS2 config) in the Qubole to use Waggle Dance endpoint.

Thanks 

Pradeep 

To view this discussion on the web, visit https://groups.google.com/d/msgid/waggle-dance-user/CANsgGk39VafC5HkqQnKXkjJSe4bFNTWaZF%2B8gYH9raYACQUdKQ%40mail.gmail.com.

[Audrey Y]

Hi Patrick and Pradeep,

Thank you for your reply! 

I have another question, I wonder if waggle dance could be connected from qubole via aws private link? The current setup is that the waggle dance service is running on an aws ec2 instance, and I am trying to use waggle dance on qubole. In this case, I setup a privatelink between qubole bastion node and the waggle dance ec2 instance so that when I connect to waggle dance on qubole, I connect to the vpc endpoint of the privatelink instead. Currently the privatelink is setup successfully but I cannot connect to waggle dance from qubole. I wonder if this is a valid approach.

Any insight would be appreciated Thank you in advance!

Thanks,

Audrey