Master / Minion connection issues

[Jesus Padro]

I have an issue where I create a distributed IDS deployment which includes a Master and only 1 Minion. I start with a fresh install of both devices. Once both are properly configured (disk partitioning includes a beefy /nsm) and installed I then run the Master setup (sosetup) via the GUI and then I run a Heavy Minion setup (sosetup) via the GUI which pairs the two. I give it about an hour of uptime then I log into my analyst system, fire up Kibana and voila all is right in the world and it is working. All the Kibana dashboards come up and for the most part display information, with the exception of the firewall. Squert and Squil also function. Give it another hour and Kibana no longer displays data within any dashboard if I look to the attached devices I see 0 sensors and 2 devices.

I get onto the Master and perform sudo salt-key -L it displays the minion's name, I perform a sudo salt '*' test.ping it returns device not reachable. I login to the minion and perform sudo service salt-minion restart, then sudo salt-minion status to check if the minion is active and it is. I then attempt to ping the master from the minion and NO GO, it returns Master not available ensure it is running. I connect to the master and perform sudo service salt-master status and it is active. I look at the firewall on the Master and the entry to allow the minion to connect to TCP Ports 4505 and 4506 is listed within sudo ufw status.

Now it is time to remove/unpair the minion using the salt-key -d (minion ID), restart the the master with sudo salt-master restart to ensure that the minion key has been removed, and reboot the master. While the master is rebooting I then reboot the Minion. Once the master completes rebooting I then login and wait for the Minion to complete rebooting. I then perform sudo salt-key -L and the Minion appears in the Unaccepted Keys, I accept it using sudo salt-key -A. I perform a ping of the Minion using sudo salt '*' test.ping and it returns True. I bring up Kibana look at the attached devices section and it shows 0 sensors and 2 devices still. I log into the minion and attempt to ping the master and NO GO.

I have rebuilt from scratch on both Master/Minion three times with the same exact results, any insight on what the cause may be. I took into consideration network latency on the first go around, so I moved the master out of the basement datacenter and moved it up to the third floor datacenter so both the Master / Minion pair are now on the same segment for initial testing prior to my roll out and integration of a Master with 13 Minions.

Thanks
Jesus

[Jesus Padro]

I am using SO 16.04.4.1

[Wes Lambert]

Hi Jesus,

When you say 'rebuilt from scratch' are you referring to re-running sosetup or re-installing from the ISO on all boxes?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Jesus Padro]

Wes
I actually did both, sosetup and bare metal reinstall. Yesterday I blew away both the Master and the Minion and started from scratch, fresh install on both, ran soup on both and rebooted. I performed sosetup on the Master first, let it perk for about half hour. Then I worked on the Minion (heavy sensor) I enabled elastic for cross cluster, once again gave it some time it to perk, then I connected to the Master via the analyst console brought up the Kibana dashboard and it seems to be working although a bit slow to display. I do get the Red Bar notification across the top of the dashboard. I acknowledge both notifications and refresh the browser and it comes up. So far all looks good, I will attempt to bring up the next Minion.

I would like to ask a question as it pertains to the HOME_NET and networks.cfg variables for Suricata and Bro respectfully. I will be deploying Minions across my network and to several remote locations out of the country. I want each Minion to monitor it respective subnet and not the overall /16 in which the entire network resides. Since suricata was installed on the Master via sosetup, would I make the config edits for each Minions instance of suricata on the Master? If so, then where do those configs reside?

Thanks
Jesus

[Wes Lambert]

Hi Jesus,

The suricata config is per-interface, so there would be a file for each interface on each sensor.

What message does the red bar provide?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Jesus Padro]

Wes
I just brought up another Minion so the count is now 2 Sensors, 4 Devices But now when I log into Kibana the dashboard is no longer providing details.

I performed a salt-key -L both Minions are listed, I then attempt to ping them via salt below is the output.

root@cs-hog-mgr:/home/jpadro# salt '*' test.ping
cs-hog-mgr:
True
cs-hog-rpg02:
True
cs-hog-peri:
Minion did not return. [No response]

As for the message that I am seeing within the Red Banner

Request Timeout after 30000ms
Error: Request Timeout after 30000ms
at https://137.228.49.51/bundles/vendors.bundle.js?v=16627:111:163257
at https://137.228.49.51/bundles/vendors.bundle.js?v=16627:111:163678

15 Visualize: Request Timeout after 30000ms
Error: Request Timeout after 30000ms
at https://137.228.49.51/bundles/vendors.bundle.js?v=16627:111:163257
at https://137.228.49.51/bundles/vendors.bundle.js?v=16627:111:163678

[Jesus Padro]

Wes
So I reran the sosetup on the Minion (peri) and performed the following;
root@cs-hog-mgr:/home/jpadro# salt-key -L
Accepted Keys:
cs-hog-mgr
cs-hog-peri
cs-hog-rpg02
Denied Keys:
Unaccepted Keys:
Rejected Keys:

root@cs-hog-mgr:/home/jpadro# salt '*' test.ping
cs-hog-mgr:
True
cs-hog-rpg02:
True
cs-hog-peri:

True

I brought up the Kibana dashboard and it displays 1 Sensor 3 Devices. So although I can ping the (peri) Minion the only data being presented on the dashboards if from the other Minion.

Still poking at it.

[Wes Lambert]

Are you running forward nodes or heavy nodes?

What is the output of the following from the master?

curl localhost:9200/_cluster/settings

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Jesus Padro]

Wes

All nodes are Heavy

Output from provided curl command
{"persistent":{"search":{"remote":{"cs-hog-peri":{"skip_unavailable":"true","seeds":["172.18.0.1:50002"]},"cs-hog-mgr":{"seeds":["127.0.0.1:9300"]},"cs-hog-rpg02":{"skip_unavailable":"true","seeds":["172.18.0.1:50001"]}}}},"transient":{}}root@cs-hog-mgr:

[Jesus Padro]

Checking for active connections to Minions

tcp 0 0 localhost:4505 localhost:35268 ESTABLISHED
tcp 0 0 cs-hog-mgr:4505 cs-hog-peri:59080 ESTABLISHED
tcp 0 0 cs-hog-mgr:4505 cs-hog-rpg02:49906 ESTABLISHED
tcp 0 0 localhost:35268 localhost:4505 ESTABLISHED

Still only one Minion is providing sensor data. 1 Sensor 3 devices.

[Jesus Padro]

sostat file on the flapper.

[Wes Lambert]

Hi Jesus,

It looks like both Elasticsearch and Logstash have failed to start on the affected heavy node.  You'll want to consult the logs in /var/log/elasticsearch/<hostname>.log and /var/log/logstash/logstash.log for clues.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Jesus Padro]

Wes
Decided to perform an soup which completed and showed a failure the the elastocsearch container

so-elasticsearch: fc765b112a6116057974cff905f3257d4756025a5324d730443c1e1672b0fb40
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/etc/elasticsearch/elasticsearch.yml\\\" to rootfs \\\"/var/lib/docker/overlay2/3ac135a3d55c78d29b2d5b7ad4abd15d96eb8fba00c1490f65ba1f0c23c8ea0d/merged\\\" at \\\"/var/lib/docker/overlay2/3ac135a3d55c78d29b2d5b7ad4abd15d96eb8fba00c1490f65ba1f0c23c8ea0d/merged/usr/share/elasticsearch/config/elasticsearch.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
so-logstash: db36ded457eeb4c5820cef7aadc50eade13724c7c5e4637221868d5449ac224b
so-curator: Already started!

What would be the best approach to correct this issue?

For now I used so-stop to on this heavy node

Thanks
Jesus

[Jesus Padro]

Wes
I found the culprit, it turns out that I had a corrupted xml file within /etc/elasticsearch it was the elasticsearch.xml. So I copied and edited the one from the other Minion and deleted the offending xml and the directory that it created to store that offending xml. I created the new one and restarted elasticsearch using so-elasticsearch-start. I observed the startup and notification that all systems green. I then started logstash using so-logstash-start and it came up in a clean state waited a few minutes and then sostat to verify and it all came up green.

Now the nest item to tackle is the transfer of data from that Minion to the master. Since when I bring up Kibana it still shows 1 sensor and 3 devices. So this sensor is still not sending data to the master. Salt shows all green since I can successfully perform salt '*' test.ping and all devices respond. The three devices display are the Master, 1 Minion and localhost. So this device although connected via salt is not represented on the dashboard.

Now it seems to be complaining about storage space space on the /nsm partition. That partition is 1.5Tb and I assigned elasticsearch 780G I may change this value down to 297G so it matchs the other Minion.

Is there a method by which I can clear the elasticsearch partition without killing this install? I believe this may be the reason that it is not displying on the master Kibana dashboard.

[Wes Lambert]

Hi Jesus,

You could try running so-elastic-reset to reset ES indices for a particular node.

I would also make sure that /root/.ssh/securityonion.ssh.conf has the appropriate values to populate the AutoSSH tunnel on the heavy node and that ES and the AutoSSH tunnel are up on each heavy node.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.