Multiple moloch clusters + MultiES issue

[Umesh Sollaprua]

I have been trying to setup multiple moloch clusters(with 1 viewer to access all clusters) for sometime now. I am not sure if im having the correct configuration or i might be missing a key step. Need help -

Servers Details -

ESCluster - 10.10.10.1 (3 nodes with 1 master)

Moloch Cluster1 - (Capture, Viewer + MultiES)

                  Starting 3 processes as below -

                          /usr/local/bin/node viewer.js -n moloch-test-1 -c ${TDIR}/etc/config.ini > ${TDIR}/logs/viewer.log 2>&1

                          /usr/local/bin/node viewer.js -n all -c ${TDIR}/etc/config.ini > ${TDIR}/logs/viewer-all.log 2>&1

                          /usr/local/bin/node multies.js -n all -c ${TDIR}/etc/config.ini > ${TDIR}/logs/multi-viewer.log 2>&1

Moloch Cluster2 - (Capture + Viewer)

                   Starting 1 viewer -

                           /usr/local/bin/node viewer.js -n moloch-test-3 -c ${TDIR}/etc/config.ini > ${TDIR}/logs/viewer.log 2>&1

I get the following error when i try to start second viewer instance on Cluster1 -

ERROR - Issue with index 'stats' make sure 'db/db.pl <eshost:esport> init' has been run { [Error: [index_not_found_exception] no such index, with { resource.type=index_or_alias resource.id=MULTIPREFIX_stats index=MULTIPREFIX_stats }]

Noticed that running 'db/db.pl <eshost:esport> init' is initializing indices for first viewer instance but is not creating indices for MULTIPREFIX

Here is my config.ini on primary moloch cluster running MultiES -

[default]

elasticsearch=http://10.10.10.1:9200

pcapDir=/data/moloch/raw

certFile=/data/moloch/etc/moloch.crt

keyFile=/data/moloch/etc/moloch.key

passwordSecret = xxx

httpRealm = Moloch

geoipFile = /data/moloch/etc/GeoIP.dat

geoipASNFile = /data/moloch/etc/GeoIPASNum.dat

rirFile = /data/moloch/etc/ipv4-address-space.csv

parsersDir = ../capture/parsers;parsers

pluginsDir = plugins;../tests/plugins;../capture/plugins

smtpIpHeaders=X-Originating-IP:;X-Barracuda-Apparent-Source-IP:

spiDataMaxIndices=-1

parseQSValue=true

parseCookieValue=true

viewPort=8005

packetThreads=2

[moloch-test-1]

pcapWriteMethod=simple

interface=eno33554944

prefix=test1

passwordSecret= xxx

[all]

viewPort=8006

passwordSecret= xxx

#   ES is running at 10.10.10.1:8200 but doesnt seem to have the right indices.

elasticsearch=http://10.10.10.1:8200

multiES=true

multiESPort=8200

multiESNodes=10.10.10.1:9200,prefix:test1;10.10.10.1:9200,prefix:test3

[Andy Wick]

* You can do whatever you want :) but when talking about "multiviewer" that is your viewer.js -n all but you save the log file for multies to multi-viewer.log, don't get confused.  I would rename that to multies.log

* I wouldn't even worry about getting 2 clusters to work until 1 cluster works, change multiESNodes to just "10.10.10.1:9200,prefix:test1"

* You should start multies before multi viewer or it won't work.  The multiviewer talks to multies talks to real ES

* I don't understand your comment "#   ES is running at 10.10.10.1:8200 but doesnt seem to have the right indices.".  ES should be running at 9200, multies should be running at 8200, hopefully that is what you meant.

* Try a simple "curl http://10.10.10.1:8200" it should return info about your first cluster AND you should see something in your multies log file (so multi-viewer.log to you)

* Try "curl http://10.10.10.1:8200/MULTIPREFIX_stats/_stats", same as above.

 

If those 2 commands don't work there is something with your ips and where you are running things.

--
You received this message because you are subscribed to the Google Groups "Moloch Full Packet Capture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moloch-fpc+...@googlegroups.com.
To post to this group, send email to moloc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Umesh Nirvani]

Got it now. Thanks. Multies and master viewer are running now with 2 moloch clusters.