Connecting elasticsearch cluster to PIO

[Maaz Mohamedy]

I have all PIO services running locally, but I am trying to get Elasticsearch to now run in a single node cluster. I am using a docker container for Elasticsearch (https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html).

I am not understanding how I should connect the remote elasticsearch (running in a docker container) to PIO. I have set the right configurations in pio-env.sh. I think I am starting up elasticsearch correctly in the docker container by running ./bin/elasticsearch -d -p <pid> (where pid is from eventserver.pid in the PredictionIO-0.12.0-incubating directory on my machine). 

I know it should be as simple as running elasticsearch from the docker container, and then binding the port of the docker container to PIO, but I'm not sure how to do this. I've been playing around with this for a couple hours now, but no luck so far.

Any help would be greatly appreciated.

[Pat Ferrel]

I have mixed feelings about containers. I absolutely think they make things better and are easier to use. My mixed feelings come from the fact that they add a completely new way of thinking about deployment and provisioning that you have to master before using them. The other downside is that only the docker community managed ones can be trusted out of the box, next trustworthy are the ones maintained by the SW owner (Elastic in this case) the rest have to be examined very closely and that requires a high degree of Docker knowledge as well as understanding the SW in the container. 

So we use out own forked containers built mostly from Docker community ones. But we do not support them because often people use them before understanding Docker. 

This logic is even more in play when the container is from an unknown source. So we flat out do not support containers in this forum unless another community member is willing to do so. I encourage you to learn Docker because it is the future but can’t help much here. You might try the Docker community if it’s a community supported container.

That said, make sure pio-env.sh points to the REST client on port 9200 of the container IP address. Also in the engine.json you should have a line in sparkConf that says something like es.nodes=“node1,node1” for whatever your container IP address(es) is. Make sure ES is listening on the IP and port by accessing the REST API with a tool like Elasticsearch Head, a Chrome plugin.

If `pio status` and `pio app list` work correctly the the problem must be in engin.json. Oh, and upgrade to PIO 0.12.1. It is the first non-incubating release and uses the ES REST client exclusively. I was unaware that the latest UR would run with 0.12.0-incubating. 

--
You received this message because you are subscribed to the Google Groups "actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-use...@googlegroups.com.
To post to this group, send email to action...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/7c91714f-eb88-4203-9998-74c0565bc37d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maaz Mohamedy]

Thank you so much for your response. I hear you and I understand what you are saying, but this is the route we have chosen to take.

I have update sparkConf in egine.json. Thanks for that info. I've made sure in pio-env.sh that PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS points to the ports where the docker container is running (9200,9300). I've also set PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME to the cluster name in the docker container. 

But now how do I make PIO listen and bind to the port of elasticsearch? Is it necessary that I have to use a tool like the chrome extension? I will upgrade pio to 0.12.1

[Maaz Mohamedy]

Can you please elaborate on what the Elasticsearch Head does and how I can use it to connect to PIO? I have elasticsearch running on the docker container and I see it on Elasticsearch Head Chrome extension.

[Pat Ferrel]

Elasticsearch head is only for Elasticsearch. I suggested it as a way to make sure ES is running correctly. If it is, read the rest of my comments.

To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/028e8fbd-fd3b-4432-ad94-067016faef9b%40googlegroups.com.

[Maaz Mohamedy]

Got it

[Maaz Mohamedy]

So I tried your suggestions, but I am still not able to get Elasticsearch (running from the docker container) to connect to PIO. 

I'm using Elasticsearch 6.3.2.. Do you think this may be causing the problem? I know only 5.x is supported, but unfortunately all of the 5.x containers don't work correctly.

[Pat Ferrel]

As I said, 6.x has not been tested so I unfortunately have no idea whether it is causing the problem or something about the container. I suppose you know that containers have IP addresses and need to be told to open certain ports?  

From: Maaz Mohamedy <maaz.m...@gmail.com>
Date: August 14, 2018 at 1:04:48 AM
To: actionml-user <action...@googlegroups.com>
Subject:  Re: Connecting elasticsearch cluster to PIO

So I tried your suggestions, but I am still not able to get Elasticsearch (running from the docker container) to connect to PIO. 

I'm using Elasticsearch 6.3.2.. Do you think this may be causing the problem? I know only 5.x is supported, but unfortunately all of the 5.x containers don't work correctly.

On Tuesday, August 14, 2018 at 12:53:41 AM UTC-7, Maaz Mohamedy wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/17f7c908-1e2e-4769-bc43-912210fa729f%40googlegroups.com.