Multiple elasticsearch nodes on same server

[KB]

I'm trying to migrate from the default single host install to a single host with multiple elasticsearch nodes running.  I have a host with 128GB of memory and I'm trying to run 3 elastic search nodes with 28GB each.  I'm running into some problems.  Is trying to run multiple nodes on the same server a dumb thing to do?

I made modifications from what I gleaned from the Installing Elasticsearch section of the README.rst and the Multiple Host HOWTO.  I changed the following (not sure about the zen.ping setting:

node.max_local_storage_nodes: 3

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 3

discovery.zen.ping.unicast.hosts: ["localhost"]

I tried both network.bind_host set to 0.0.0.0, the actual IP address, and 127.0.0.1. 

Head shows "cluster health: not connected" and connecting to port 9200 show status code 503.

Have you had success doing this?

What did appear to work is:

I created 3 elasticsearch.yml files, elasticsearch-host1.yml, elasticsearch-host2.yml, elasticsearch-host3.yml:

All files used the same cluster.name.

I left the following with their defaults from single host install:

node.max_local_storage_nodes: 1

gateway.recover_after_nodes: 1

gateway.expected_nodes: 1

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.unicast.hosts: ["localhost"]

It didn't seem to like writing to the same path.logs directory so I modified the 3 elasticsearch configs to have:

path.data: /data/moloch/data/host1

path.logs: /data/moloch/logs/host1 

path.data: /data/moloch/data/host2

path.logs: /data/moloch/logs/host2

path.data: /data/moloch/data/host3

path.logs: /data/moloch/logs/host3 

I left networking at 127.0.0.1. 

I created 3 run_es.sh files using the different hostnames and config files.  Doing this Head shows the cluster green with 3 nodes, node info shows what appears to be a balanced document distribution, and Moloch viewer/capture appear to be working correctly.  I'm not sure what I'm doing at this point.  Any help or suggestions would be greatly appreciated.

[Andy]

We run multiple nodes per host and use a single yml file, since I don't want to have multiple config files. :)

You need to set node.max_local_storage_nodes  to the number of nodes, but I think that only matters if all the nodes use the same path for data.

We also just comment out the bind host, which defaults to 0.0.0.0

The easier fix for logging is to change logging.yml where it has ${cluster.name} to ${node.name}

Also if you really want to have different paths, you can change your yml files to reference a environment variable by just using ${ENVNAME} that you set in your run_es.sh script

Andy

[KB]

Works like a charm.  Thank you for your help!

[Xinity]

hy Andy,

i've tried what you advised but it failed, i still have one data/master node and a client node.

i haven't found how to start several data nodes on the same machine using the init script provided in the .deb file

FYI: i'm uing 0.90.10 version

my elasticsearch.yml contains only this:

cluster.name: MyMonitor

network.host: 127.0.0.1

node.name: "DataMonitor01"

path.data: /home/elasticsearch

discovery.zen.ping.multicast.enabled: false

node.max_local_storage_nodes: 5

any clue ?

Regards,

[Andy]

Are there any log files?  I've never used the deb script so I don't know exact details, but you can not set the node name since they need to be unique.  

What we do is set the node name manually before starting ES.  So in our yml we have

node.name: "${ES_HOSTNAME}"

and then in our start up script we have

export ES_HOSTNAME=`hostname -s`a

ES_HEAP_SIZE=22G bin/elasticsearch -Des.config=/srv/moloch/elasticsearch.yml

export ES_HOSTNAME=`hostname -s`b

ES_HEAP_SIZE=22G bin/elasticsearch -Des.config=/srv/moloch/elasticsearch.yml

There might be a better way these days. :)

Also I don't think you want network.host to be 127.0.0.1 still, probably should be 0.0.0.0

Andy

[rich.b...@gmail.com]

Hi Xinity,

I'm sure you have long found your answer by now, but in case someone like me is reading this post...

In addition to the above settings (actually, I didn't explicitly set network.host nor mode.max_local_storage_nodes, I left the defaults) I set:

discovery.zen.ping.unicast.hosts: ["127.0.0.1[9300-9302]"]

I should also mention that I also created separate config files for each of my nodes. And I ran 3 instances of ES.

Instance 1 (master, no data):

./bin/elasticsearch -Des.node.data=false -Des.node.master=true -Des.node.name=NoData

(this acted as the master node and overrode the three settings listed)

Instance 2 (data):

 ./bin/elasticsearch -Des.config=/usr/local/opt/elasticsearch/config/elasticsearch.0.yml

(also changed path.data to /usr/local/var/elasticsearch/0/ and path.logs to /usr/local/var/log/elasticsearch/0/)

Instance 3 (data):

 ./bin/elasticsearch -Des.config=/usr/local/opt/elasticsearch/config/elasticsearch.1.yml

(also changed path.data to /usr/local/var/elasticsearch/1/ and path.logs to /usr/local/var/log/elasticsearch/1/)

I left the original elasticsearch.yml used by the master (instance 1) with all the defaults (other than cluster name)

For elasticsearch.0.yml and elasticsearch.1.yml, I changed the settings as described above (and obviously the cluster name, so that all three config files had the same cluster name).

Oh, I had also enabled multicasting on my machine, but I'm not sure if that mattered since I am apparently using unicasting. I am new to all this, so I'm not quite sure what the story behind all that is.

Hope this helps.

Best,

Rich

[Andy]

Definitely can use separate config files, you can also use env vars and just 1 config file (which I recommend in case you need to make more changes later)

Here is a sample where we run a master on one node on one machine, for this cluster there are 2 nodes on each of 10 machines.  You could also add more variables for logs and data path.

#!/bin/sh

if [ "$(hostname -s)" == "moloches-master-machine" ]; then

  export ES_MASTER=true

else

  export ES_MASTER=false

fi

export ES_HOSTNAME=`hostname -s`a

ES_HEAP_SIZE=30G bin/elasticsearch -Des.config=/srv/moloch/elasticsearch.yml -d

if [ "$ES_MASTER" == "true" ]; then

sleep 10

fi

export ES_MASTER=false

export ES_HOSTNAME=`hostname -s`b

ES_HEAP_SIZE=30G bin/elasticsearch -Des.config=/srv/moloch/elasticsearch.yml -d

then in your config file you have

node.name: "${ES_HOSTNAME}"

node.master: "${ES_MASTER}"