How to add support for REST-based Elasticsearch clients to titan-es ?

[Ingo Jaeckel]

I started looking at the titan-es code to find a way to integrate something like JEST. In addition to local and transport (TCP) mode, I would like to support communication over ES' HTTP interface. That would allow users to run Titan against the managed AWS Elasticsearch service which only provides a HTTP(S) interface. See this post for more information.

I am looking for some feedback on the approach I am considering so far:

* Create a new class "ElasticsearchHttpClient" that implements the org.elasticache.client.Client interface. The new class will use the JestClient as it's internal client. This way it will communicate with ES over HTTP. The new class will likely extend ES' AbstractClient to reduce the methods that have to be implemented to: admin(), settings(), execute(), threadPool(), and close().

* Add a new enum to ElasticSearchSetup e.g. "HTTP_CLIENT"

* Ensure that the connect() method on HTTP_CLIENT returns an instance of Connection which contains proper values for node and client. The client member would be an instance of the new ElasticsearchHttpClient class.

* Ensure that ElasticSearchIndex.interfaceConfiguration() method retrieves the correct instance of Connection (containing the new ElasticsearchHttpClient) if the INTERFACE is configured as HTTP_CLIENT. From that point on the rest of the code should continue to work over the new protocol. 

Does that sound like it should work? The 1st step is my biggest concern - I am not confident that I can implement all Client methods using the JestClient. 

Has somebody tried this before?

Thanks,

Ingo

[Jason Plurad]

Hi Ingo,

Pretty cool that you're looking into working on this! I'd be happy to test or collaborate with you on this.

What you'd need to do from a Titan perspective is implement the IndexProvider interface. My guess is that it isn't feasible to make Jest look like a full Elasticsearch client.

I think you would use JestHttpClient -- you don't need to implement the Jest interface. IndexProvider has methods to create/drop/mutate/query an index, which you should be able to do over HTTP. Check the Elasticsearch HTTP documentation to see if you can do all the required methods on IndexProvider with JestHttpClient.

There's already an ElasticSearchIndex implementation of IndexProvider, which does NODE and TRANSPORT. You're trying to add an HTTP or JEST option. So you might consider shoehorning your changes into ElasticSearchIndex, but I'm not sure how well that will work out since the 2 existing impls are both full Elasticsearch clients. Perhaps consider creating a separate ElasticSearchHttpIndex implements IndexProvider if it's cleaner.

Keep the Titan community posted with your progress.

-- Jason