Elastic search could not find some docs in a single-master cluster

[Julius Somoye]

Hi all,

I have set up a cluster in a single master mode with elastic search
installed on server nodes as directed in the search integration
documentation.

I have a 3 workstations, one Master(52.19.76.71),
Server1(50.19.161.128) and Server2(50.19.27.156). The terrastore
sides of things work perfectly well.

I have a script that randomly connects to either of the server nodes
and inserts a document. The document insert can happen on server1 and
I can retrieve it via server2 without any problem as the client does
not need to know about the partition info.

However if I try to search via ElasticSearch nothing will be returned
unless I run the search on the node through which the document got
inserted. for example I can add a doc through server1:

$ curl -v -X PUT -H "Content-Type: application/json" -d "{\"test
\" : \"test\"}" http://50.19.161.128:8205/mybucket/test

And I can retrieve the doc directly through server2:
$ curl -X GET -H "Content-Type: application/json" http://50.19.27.156:8205/mybucket/test

But I cannot get the doc through ElasticSearch query on server2:
$ curl -X GET -H "Content-Type: application/json"
http://50.19.27.156:9000/search/mybucket/_search?q=test:'test'
{"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":
0,"max_score":null,"hits":[]}}

while doc is return if searched through server1:
$ curl -X GET -H "Content-Type: application/json"
http://50.19.161.128:9000/search/mybucket/_search?q=test:'test'

{"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":
1,"max_score":0.5945348,"hits":
[{"_index":"search","_type":"mybucket","_id":"test","_score":
0.5945348, "_source" : {"test":"test"}}]}}

My terrastore-search.properties content on both server nodes is (the
network host value differs on server2);

node.data = true
http.enabled = true
http.port = 9000
transport.tcp.port = 9001
network.host = 50.19.161.128

Please has anyone experienced this before? Is there something I'm
doing wrong?

Thanks