Elasticsearch daemon problem

[Miki Alkalay]

Hi Wazuh,

today my elastic get down (no reason)

the message is:

[root@wazuh ~]# systemctl -l status elasticsearch

● elasticsearch.service - Elasticsearch

   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Wed 2019-06-26 11:13:29 IDT; 24s ago

     Docs: http://www.elastic.co

  Process: 5280 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)

 Main PID: 5280 (code=exited, status=1/FAILURE)

Jun 26 11:13:23 wazuh systemd[1]: Started Elasticsearch.

Jun 26 11:13:25 wazuh elasticsearch[5280]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Jun 26 11:13:29 wazuh systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE

Jun 26 11:13:29 wazuh systemd[1]: Unit elasticsearch.service entered failed state.

Jun 26 11:13:29 wazuh systemd[1]: elasticsearch.service failed.

please advise

Miki

[Jesús Ángel González]

Hi Miki,

Something might be wrong in your configuration files or service files.

Let’s see more details about your deployment:

• Journal for the user elasticsearch

journalctl -u elasticsearch > /tmp/output.txt && cat /tmp/output.txt

• Logs from Elasticsearch

tail -80 /var/log/elasticsearch/<elasticsearch|cluster-name>.log

• Configuration file for Elasticsearch

cat /etc/elasticsearch/elasticsearch.yml

That’s all, once we can review the above commands we can continue helping you properly.

Regards,
Jesús

​

[Miki Alkalay]

[root@wazuh ~]# journalctl -u elasticsearch > /tmp/output.txt && cat /tmp/output.txt
-- Logs begin at Wed 2019-06-26 11:09:35 IDT, end at Wed 2019-06-26 11:58:58 IDT. --
Jun 26 11:09:51 wazuh systemd[1]: Started Elasticsearch.
Jun 26 11:09:56 wazuh elasticsearch[3423]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 26 11:10:02 wazuh systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 26 11:10:02 wazuh systemd[1]: Unit elasticsearch.service entered failed state.
Jun 26 11:10:02 wazuh systemd[1]: elasticsearch.service failed.

Jun 26 11:13:23 wazuh systemd[1]: Started Elasticsearch.
Jun 26 11:13:25 wazuh elasticsearch[5280]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 26 11:13:29 wazuh systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 26 11:13:29 wazuh systemd[1]: Unit elasticsearch.service entered failed state.
Jun 26 11:13:29 wazuh systemd[1]: elasticsearch.service failed.

Jun 26 11:20:22 wazuh systemd[1]: Started Elasticsearch.
Jun 26 11:20:23 wazuh elasticsearch[7582]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 26 11:20:26 wazuh systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 26 11:20:26 wazuh systemd[1]: Unit elasticsearch.service entered failed state.
Jun 26 11:20:26 wazuh systemd[1]: elasticsearch.service failed.

[root@wazuh ~]# tail /var/log/elasticsearch/elasticsearch.log elastic.log
==> /var/log/elasticsearch/elasticsearch.log <==
[2019-06-13T15:15:01,976][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.06.13]
[2019-06-13T15:30:01,778][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.06.13]
[2019-06-13T15:45:01,762][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.06.13]
[2019-06-13T15:59:32,628][INFO ][o.e.n.Node               ] [wazuh] stopping ...
[2019-06-13T15:59:32,661][INFO ][o.e.x.w.WatcherService   ] [wazuh] stopping watch service, reason [shutdown initiated]
[2019-06-13T15:59:32,803][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wazuh] [controller/22120] [Main.cc@148] Ml controller exiting
[2019-06-13T15:59:32,804][INFO ][o.e.x.m.p.NativeController] [wazuh] Native controller process has stopped - no new native processes can be started
[2019-06-13T15:59:33,108][INFO ][o.e.n.Node               ] [wazuh] stopped
[2019-06-13T15:59:33,109][INFO ][o.e.n.Node               ] [wazuh] closing ...
[2019-06-13T15:59:33,167][INFO ][o.e.n.Node               ] [wazuh] closed

[root@wazuh ~]# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: Spartan
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: Wazuh-Spartan1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
# network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
discovery.type: single-node
[root@wazuh ~]#

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e39efc08-c776-401e-9571-3438781e1406%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards

Miki Alkalay

Mobile: 972-54-6496293

[Jesús Ángel González]

Hi Miki,

I think it’s related to bootstrap.memory_lock: true, can you temporary comment/remove that line from
your elasticsearch.yml and restart the service?

If the node starts properly removing that line then we can restore it and review all other steps needed for that setting,
but in the first place, I want to discard any error related to that setting.

Regards,
Jesús

​

[Miki Alkalay]

Hi,

I comment it out and still having this issue

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/95508e8d-2bfb-490f-9190-d7cecece07be%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

I think now that the OOM (out of memory) killer of Linux is killing your service, can we check system logs?

grep -iR 'killed process' /var/log

This command is also helpful here:

free -h

Regards,
Jesús

​

[Miki Alkalay]

[root@wazuh ~]# grep -iR 'killed process' /var/log
[root@wazuh ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:            25G        1.0G         19G        8.4M        4.5G         23G
Swap:            0B          0B          0B

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d51f324d-2191-4219-aa0b-6de2e2b6b07e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hello again Miki,

Let’s hardcode the ulimit value for open files:

ulimit -n 65536         
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

Now, restart the service, and look again for the logs:

systemctl daemon-reload
systemctl restart elasticsearch

Leave a CLI opened using this command:

tail -f /var/log/elasticsearch/elasticsearch.log

On the other hand, just some more questions:

• Are you using Docker?
• Have you modified some other files such as the Elasticsearch service or some other critical files?

Regards,
Jesús

​

[Miki Alkalay]

Hi,

Did what you asked me.

I'm not using any dockers.

yesterday it worked.

same error:

[root@wazuh ~]# ulimit -n 65536
[root@wazuh ~]# echo "* soft nofile 65536" >> /etc/security/limits.conf
[root@wazuh ~]# echo "* hard nofile 65536" >> /etc/security/limits.conf
[root@wazuh ~]# systemctl daemon-reload
[root@wazuh ~]# systemctl restart elasticsearch
[root@wazuh ~]# tail -f /var/log/elasticsearch/elasticsearch.log

[2019-06-13T15:15:01,976][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.06.13]
[2019-06-13T15:30:01,778][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.06.13]
[2019-06-13T15:45:01,762][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.06.13]
[2019-06-13T15:59:32,628][INFO ][o.e.n.Node               ] [wazuh] stopping ...
[2019-06-13T15:59:32,661][INFO ][o.e.x.w.WatcherService   ] [wazuh] stopping watch service, reason [shutdown initiated]
[2019-06-13T15:59:32,803][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wazuh] [controller/22120] [Main.cc@148] Ml controller exiting
[2019-06-13T15:59:32,804][INFO ][o.e.x.m.p.NativeController] [wazuh] Native controller process has stopped - no new native processes can be started
[2019-06-13T15:59:33,108][INFO ][o.e.n.Node               ] [wazuh] stopped
[2019-06-13T15:59:33,109][INFO ][o.e.n.Node               ] [wazuh] closing ...
[2019-06-13T15:59:33,167][INFO ][o.e.n.Node               ] [wazuh] closed

[root@wazuh ~]# systemctl -l status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Wed 2019-06-26 13:31:19 IDT; 854ms ago
     Docs: http://www.elastic.co
  Process: 19654 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 19654 (code=exited, status=1/FAILURE)

Jun 26 13:31:13 wazuh systemd[1]: Started Elasticsearch.
Jun 26 13:31:14 wazuh elasticsearch[19654]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 26 13:31:19 wazuh systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 26 13:31:19 wazuh systemd[1]: Unit elasticsearch.service entered failed state.
Jun 26 13:31:19 wazuh systemd[1]: elasticsearch.service failed.

Miki

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9251d3bf-1805-4102-b8fe-3a8e07a9368b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Miki Alkalay]

Hi,

Sorry for the pressure, any update, my manager is down

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9251d3bf-1805-4102-b8fe-3a8e07a9368b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

Pretty weird, we are not looking at the right place I think.

The service is being killed but I don’t know who and why yet.

Add this line to your logging settings:

echo 'rootLogger.level = debug' >> /etc/elasticsearch/log4j2.properties

Now restart Elasticsearch.

Now if you can send me a file with the content of at least 200 lines of your Elasticsearch log it would be helpful.

With the above modification the log file will be more verbose, so please after your node is crashed again, execute the next command:

tail -200 /var/log/elasticsearch/elasticsearch.log > /tmp/output.txt

Then, send me the /tmp/output.txt content or the file so I can look for other technical logs that may help here.

Regards,
Jesús

​

[Miki Alkalay]

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2fef093c-7515-42b4-adee-a9bcc82ef3b7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

Still not clear, it just says something/someone killed the service with no reason.

I’ve also tried with your exact configuration with no luck… (it worked for me).

Please, give us the output from the next command (/tmp/filtered.txt content):

journalctl -xb > /tmp/output-journal.txt
cat /tmp/output-journal.txt | grep -i -E "score|kill|elasticsearch" > /tmp/filtered.txt
rm -f /tmp/output-journal.txt

Please, send us the content of /tmp/filtered.txt, it may show something about the service killer.

Best regards,
Jesús

​

[Miki Alkalay]

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f2e29794-68e2-4378-95b9-d1a56ed09af3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hello again Miki,

I’m still thinking about your problem, I could not reproduce your issue even using the same configuration.

Please, can you edit the Elasticsearch service? just for debug purposes.

Execute this:

systemctl edit elasticsearch

Add this content, then save and exit:

[Service]
LimitMEMLOCK=infinity

Now, please reload the service:

systemctl daemon-reload
systemctl restart elasticsearch

Also, paste the output of the next command just after restarting Elasticsearch and before it dies, please:

ps aux | grep java

So we can check how Elasticsearch is launched (the java command line being used).

Sorry about having too many questions but I could not reproduce the issue and its reason is not clear at all.

Regards,
Jesús

​

[Miki Alkalay]

[root@wazuh ~]# ps aux | grep java
logstash  6943  211  2.3 3690064 625180 ?      SNsl 09:10   0:52 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.8.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
elastic+  7050  138  8.5 5004864 2261080 ?     Ssl  09:10   0:09 /usr/share/elasticsearch/jdk/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-11854605583245122247 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=1073741824 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root      7197  0.0  0.0 112708   976 pts/0    S+   09:10   0:00 grep --color=auto java

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/51aa92fb-c9f5-414d-88a2-8f520e2912a6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

Sorry if I’m being repetitive, but I need to confirm some steps from our previous messages:

1. Have you added debug logging level to Elasticsearch? I can’t see debug messages in your logs.
2. Have you edited the Elasticsearch service file as I said?
3. Is still failing?
4. Have you modified any other important file that we should know about it?

Regards,
Jesús

​

[Miki Alkalay]

it's still failing

i didn't change a thing in the files.

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/cc1c2280-78e1-4b36-ae03-4c24a756955e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hello again Miki,

After some messages with no results, please let me know about the next details so we can build an environment like yours. We’ve already done it, but we may differ at some steps.

• Exact installed package

// RPM based
rpm -qa elasticsearch

// Deb based
dpkg -l elasticsearch

• OS details

cat /etc/os-release

• Permissions

ls -lh /etc/elasticsearch
ls -lh /usr/share/elasticsearch
ls -lh /var/lib/elasticsearch
ls -lh /tmp

• Disk usage

df -h

• Instance details

lscpu

Thanks for your patience.

On the other hand, if you did the logger modifications, is there any debug log in /var/log/elasticsearch/<elasticfile>.log? they should include [DEBUG] in the logline.

Regards,
Jesús

​

[Miki Alkalay]

Hi,

I have 2 servers with the same problems.

i think it's related to some upgrades or updates because the Elastic repository was enables

[root@wazuh ~]# rpm -qa elasticsearch
elasticsearch-7.1.1-1.x86_64
[root@wazuh ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@wazuh ~]# ls -lh /etc/elasticsearch
total 68K
-rw-rw----. 1 root elasticsearch  199 Jun 27 10:42 elasticsearch.keystore
-rw-rw----. 1 root elasticsearch 2.8K May 23 17:15 elasticsearch.yml
-rw-rw----. 1 root           993 2.9K Jun 26 12:22 elasticsearch.yml.rpmsave
-rw-rw----. 1 root elasticsearch 3.6K May 23 17:15 jvm.options
-rw-rw----. 1 root           993 3.6K Jun 26 13:02 jvm.options.rpmsave
-rw-rw----. 1 root elasticsearch  17K May 23 17:15 log4j2.properties
-rw-rw----. 1 root           993  17K Jun 26 15:41 log4j2.properties.rpmsave
-rw-rw----. 1 root elasticsearch  473 May 23 17:15 role_mapping.yml
-rw-rw----. 1 root elasticsearch  197 May 23 17:15 roles.yml
-rw-rw----. 1 root elasticsearch    0 May 23 17:15 users
-rw-rw----. 1 root elasticsearch    0 May 23 17:15 users_roles
[root@wazuh ~]# ls -lh /usr/share/elasticsearch
total 480K
drwxr-xr-x.  2 root root 4.0K Jun 27 10:42 bin
drwxr-xr-x.  8 root root   96 Jun 27 10:42 jdk
drwxr-xr-x.  3 root root 4.0K Jun 27 10:42 lib
-rw-r--r--.  1 root root  14K May 23 17:01 LICENSE.txt
drwxr-xr-x. 29 root root 4.0K Jun 27 10:42 modules
-rw-rw-r--.  1 root root 437K May 23 17:06 NOTICE.txt
drwxr-xr-x.  2 root root    6 May 23 17:15 plugins
-rw-r--r--.  1 root root 8.3K May 23 17:01 README.textile
[root@wazuh ~]# ls -lh /var/lib/elasticsearch
total 1.3G
-rw-------. 1 994 993 1.3G May 29 12:20 java_pid8397.hprof
drwxr-xr-x. 3 994 993   15 Mar 13 20:38 nodes
[root@wazuh ~]# ls -lh /tmp
total 56K
-rw-r--r--. 1 root   root     15 May 29 11:55 01-wazuh.conf.bak
drwx------. 3    992    991   47 Jun  4 14:49 chromium-8A0NNp
drwx------. 3    992    991   47 Jun  4 15:22 chromium-cZu005
drwx------. 3    992    991   47 Jun  6 11:51 chromium-enwXq9
drwx------. 3    992    991   47 Jun  4 18:33 chromium-FBqsUV
drwx------. 3    992    991   47 May 30 13:08 chromium-Gge0wj
drwx------. 3 kibana kibana   47 Jun 27 11:09 chromium-hfi5c7
drwx------. 3    992    991   47 Jun 11 12:13 chromium-jmRQ97
drwx------. 3    992    991   21 Jun  1 16:29 chromium-LuHGf1
drwx------. 3    992    991   21 May 29 12:02 chromium-LZ4EiD
drwx------. 3    992    991   47 May 29 11:42 chromium-nfHcRx
drwx------. 3    992    991   47 Jun  6 11:03 chromium-rMitM7
drwx------. 3 kibana kibana   47 Jun 27 11:11 chromium-s5Hcgu
drwx------. 3 kibana kibana   47 Jun 27 10:51 chromium-S5Y8E4
drwx------. 3    992    991   47 May 30 13:06 chromium-stUujQ
drwx------. 3    992    991   47 May 29 11:35 chromium-Vwhgcn
drwx------. 3    992    991   47 Jun 13 16:10 chromium-vx3M0U
drwx------. 3    992    991   47 Jun 10 13:58 chromium-xEhuk4
drwx------. 3    992    991   47 Jun  3 11:44 chromium-XnsOsO
drwx------. 3    992    991   47 Jun  4 18:26 chromium-Ym3r2j
drwx------. 2 root   root      6 May 29 11:52 elasticsearch-11167986531354103618
drwx------. 2 root   root      6 Jun 27 10:42 elasticsearch-15275071099666715580
drwx------. 2 root   root      6 Mar 13 20:37 elasticsearch-3538527049783062487
-rw-r--r--. 1 root   root    33K Jun 27 11:27 elasticsearch.log
-rw-r--r--. 1 root   root   3.9K Jun 26 16:04 filtered.txt
drwxr-xr-x. 2    993    992    6 Jun 27 10:40 hsperfdata_logstash
drwxr-xr-x. 2 root   root      6 Jun 27 10:42 hsperfdata_root
-rw-r--r--. 1 root   root    12K Jun 26 15:41 output.txt
drwx------. 2 miki   miki      6 Mar 13 13:03 ssh-mEEPxbDaGu
drwx------. 2 miki   miki      6 Mar 23 20:12 ssh-VbFhRLdN6l
drwx------. 3 root   root     17 Jun 27 10:03 systemd-private-d23a1d614ee74dacaddc810239dcac97-chronyd.service-guDeyf
[root@wazuh ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   11G   40G  22% /
devtmpfs         13G     0   13G   0% /dev
tmpfs            13G     0   13G   0% /dev/shm
tmpfs            13G   17M   13G   1% /run
tmpfs            13G     0   13G   0% /sys/fs/cgroup
tmpfs           2.6G     0  2.6G   0% /run/user/0
[root@wazuh ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU @ 2.30GHz
Stepping:              0
CPU MHz:               2300.000
BogoMIPS:              4600.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear spec_ctrl intel_stibp arch_capabilities
[root@wazuh ~]#

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0e2b9ce5-cc86-47e8-95e2-11d9e31b9117%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

From your latest logs I can see maybe these locations are not writable.

Can we try to force the user and group for Elasticsearch related files?

chown elasticsearch:elasticsearch -R /usr/share/elasticsearch
chown elasticsearch:elasticsearch -R /etc/elasticsearch
chown elasticsearch:elasticsearch -R /var/lib/elasticsearch

Restart the service:

systemctl restart elasticsearch

Let’s see again if it works or not.

Regards,
Jesús

​

[Miki Alkalay]

Hi,

It didn't help

as I told you, I have 2 system with the same symptoms 

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b8e27cc7-0df3-4fe8-b60f-76c4e43f02a4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hello again Miki,

Is it possible that you have an orphan Java process running?

Since Elasticsearch uses the port 9200, let’s check if there is any other process using it:

netstat -nlp | grep 9200

Example output:

tcp6       0      0 172.16.1.2:9200         :::*                    LISTEN      3002/java

If that’s the case, kill it:

kill -9 3002 // using 3002 as the PID from my last output

In addition, and since your node is crashed, the next command should show nothing:

ps aux | grep 'elastic'

Regards,
Jesús

​

[Miki Alkalay]

[root@wazuh ~]# ps aux | grep 'elastic'
root     26335  0.0  0.0 112708   980 pts/0    S+   12:39   0:00 grep --color=auto elastic

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5e4abc40-1d0d-4358-8952-9d7137a4c108%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Miki Alkalay]

Hi,

Still waiting for your reply, i have 2 system that are down

can't continue 

Thanks

Miki

[Jesús Ángel González]

Hi Mike,

Since you have two instances, can you share with us your elasticsearch.yml for both instances? (I've only seen one of them)

Also, the logs for that second instance would be useful.

Regards,
Jesús

​

[Miki Alkalay]

Hi,

i just role one of them to previews version(snapshot), so now it's only one instance

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/32c4a0e3-a5b5-41b8-8043-52d60998135e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hello again Miki,

At this point, I think your node.lock is corrupted.

Since it’s safe to delete it under certain situations like yours, let’s delete it please:

rm -f /var/lib/elasticsearch/nodes/0/node.lock

Now, restart Elasticsearch:

systemctl restart elasticsearch

If it’s still failing, please paste the logs of Elasticsearch from the restart until it crashed so we can check if the deletion took effect.

Regards,
Jesús

​

[Miki Alkalay]

sorry but i'm in the same place, not solved

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e946a6d7-51ca-4949-8a7b-f7abaa21eb17%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

Even if you are in the same place, I need some logs so we can check what happened after the deletion of the node.lock.

Please, share with us the logs after the last modification.

Regards,

Jesús

[Miki Alkalay]

Hi Jesus,

Sorry it's just frustrated.

which logs do you need,

Thanks Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e3badb63-1fa5-40b2-b0c7-0d9564d96e8f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jesús Ángel González]

Hi Miki,

Same logs always, the logs under /var/log/elasticsearch/<clustername|elasticsearch>.log, otherwise, I can’t see our progress
in every modification/change we did.

Sorry about being frustrating but it’s the very first time I’ve seen this situation, it’s pretty weird.

Regards,
Jesús

​

[Miki Alkalay]

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/80613d35-ee32-4258-845f-2d5b867632c2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Miki Alkalay]

Hi,

Any news regarding this?

Miki

Get Outlook for Android

---

From: Miki Alkalay <realnet...@gmail.com>
Sent: Thursday, June 27, 2019 4:23:23 PM
To: Jesús Ángel González
Cc: Wazuh mailing list
Subject: Re: Elasticsearch daemon problem

 

[Miki Alkalay]

Hi Jesús,

I have managed to solve the problem of the Daemon problem (YML file configuration)

now the services are working fine.

but i have the error on the web --> 502 bad gateway, i assume it's something related to Kibana with Nginx.

Attached all YML files,

can you please look at it.

Thanks

Miki

[Pablo Torres]

Hi Miki,

I can see in your default.conf:

proxy_pass http://localhost:5601/;

could you please replace localhost with Kibana IP, restart nginx (systemctl restart nginx) and tell me if the problem persists?

If the problem persists, could you please send us some nginx error logs? 

These logs can be found at /var/log/nginx/nginx.error.log and /var/log/nginx/error.log.

One more thing, have you followed our documentation to use nginx? https://documentation.wazuh.com/3.7/installation-guide/optional-configurations/kibana_ssl.html

Regards,

Pablo Torres

[Miki Alkalay]

Hi,

I managed to solve the problem by reinstalling Kibana

Thanks

Miki

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0674ff9f-803c-4c2a-92a1-85c849a7e1f5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ss]

how to solve this problem 

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

[2020-05-29T12:58:41,338][INFO ][o.e.e.NodeEnvironment    ] [wazuh-node-1] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [237.8gb], net total_space [294.2gb], types [ext4]

[2020-05-29T12:58:41,348][INFO ][o.e.e.NodeEnvironment    ] [wazuh-node-1] heap size [990.7mb], compressed ordinary object pointers [true]

[2020-05-29T12:58:41,631][INFO ][o.e.n.Node               ] [wazuh-node-1] node name [wazuh-node-1], node ID [6_yt38rjR4OAtXMgd5pmHQ], cluster name [elasticsearch]

[2020-05-29T12:58:41,632][INFO ][o.e.n.Node               ] [wazuh-node-1] version[7.5.1], pid[5676], build[default/tar/3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96/2019-12-16T22:57:37.835892Z], OS[Linux/4.15.0-101-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]

[2020-05-29T12:58:41,633][INFO ][o.e.n.Node               ] [wazuh-node-1] JVM home [/home/wazuh3/elasticsearch-7.5.1/jdk]

[2020-05-29T12:58:41,634][INFO ][o.e.n.Node               ] [wazuh-node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-13257403823209684844, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/wazuh3/elasticsearch-7.5.1, -Des.path.conf=/home/wazuh3/elasticsearch-7.5.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]

[2020-05-29T12:58:45,541][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [aggs-matrix-stats]

[2020-05-29T12:58:45,542][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [analysis-common]

[2020-05-29T12:58:45,542][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [flattened]

[2020-05-29T12:58:45,543][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [frozen-indices]

[2020-05-29T12:58:45,543][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [ingest-common]

[2020-05-29T12:58:45,543][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [ingest-geoip]

[2020-05-29T12:58:45,544][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [ingest-user-agent]

[2020-05-29T12:58:45,544][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [lang-expression]

[2020-05-29T12:58:45,545][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [lang-mustache]

[2020-05-29T12:58:45,545][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [lang-painless]

[2020-05-29T12:58:45,546][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [mapper-extras]

[2020-05-29T12:58:45,546][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [parent-join]

[2020-05-29T12:58:45,547][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [percolator]

[2020-05-29T12:58:45,547][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [rank-eval]

[2020-05-29T12:58:45,548][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [reindex]

[2020-05-29T12:58:45,548][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [repository-url]

[2020-05-29T12:58:45,549][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [search-business-rules]

[2020-05-29T12:58:45,549][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [spatial]

[2020-05-29T12:58:45,550][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [transform]

[2020-05-29T12:58:45,550][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [transport-netty4]

[2020-05-29T12:58:45,551][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [vectors]

[2020-05-29T12:58:45,551][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-analytics]

[2020-05-29T12:58:45,551][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-ccr]

[2020-05-29T12:58:45,552][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-core]

[2020-05-29T12:58:45,552][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-deprecation]

[2020-05-29T12:58:45,553][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-enrich]

[2020-05-29T12:58:45,553][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-graph]

[2020-05-29T12:58:45,554][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-ilm]

[2020-05-29T12:58:45,554][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-logstash]

[2020-05-29T12:58:45,554][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-ml]

[2020-05-29T12:58:45,555][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-monitoring]

[2020-05-29T12:58:45,555][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-rollup]

[2020-05-29T12:58:45,556][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-security]

[2020-05-29T12:58:45,556][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-sql]

[2020-05-29T12:58:45,557][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-voting-only-node]

[2020-05-29T12:58:45,557][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] loaded module [x-pack-watcher]

[2020-05-29T12:58:45,558][INFO ][o.e.p.PluginsService     ] [wazuh-node-1] no plugins loaded

[2020-05-29T12:58:51,455][INFO ][o.e.x.s.a.s.FileRolesStore] [wazuh-node-1] parsed [0] roles from file [/home/wazuh3/elasticsearch-7.5.1/config/roles.yml]

[2020-05-29T12:58:52,437][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wazuh-node-1] [controller/5792] [Main.cc@110] controller (64 bit): Version 7.5.1 (Build ae3c3c51b849be) Copyright (c) 2019 Elasticsearch BV

[2020-05-29T12:58:53,227][DEBUG][o.e.a.ActionModule       ] [wazuh-node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security

[2020-05-29T12:58:53,357][INFO ][o.e.d.DiscoveryModule    ] [wazuh-node-1] using discovery type [zen] and seed hosts providers [settings]

[2020-05-29T12:58:55,270][INFO ][o.e.n.Node               ] [wazuh-node-1] initialized

[2020-05-29T12:58:55,272][INFO ][o.e.n.Node               ] [wazuh-node-1] starting ...

[2020-05-29T12:58:55,543][INFO ][o.e.t.TransportService   ] [wazuh-node-1] publish_address {172.20.12.48:9300}, bound_addresses {172.20.12.48:9300}

[2020-05-29T12:58:55,551][ERROR][o.e.g.GatewayMetaState   ] [wazuh-node-1] failed to read or upgrade local state, exiting...

org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st

        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:167) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:73) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:154) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:90) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.node.Node.start(Node.java:696) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.5.1.jar:7.5.1]

        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.5.1.jar:7.5.1]

Caused by: java.io.IOException: failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 15 more

Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=680781824 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st")))

        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:307) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 15 more

Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=680781824 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st")))

        at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]

        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]

        at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]

        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:290) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 15 more

[2020-05-29T12:58:55,575][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [wazuh-node-1] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st]; nested: IOException[failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st]; nested: CorruptStateException[org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=680781824 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st")))]; nested: CorruptIndexException[codec footer mismatch (file truncated?): actual footer=680781824 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st")))];

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.5.1.jar:7.5.1]

        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]

Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st

        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:167) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:73) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:154) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:90) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.node.Node.start(Node.java:696) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 6 more

Caused by: java.io.IOException: failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:73) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:154) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:90) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.node.Node.start(Node.java:696) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 6 more

Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=680781824 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st")))

        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:307) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:73) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:154) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:90) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.node.Node.start(Node.java:696) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 6 more

Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=680781824 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st")))

        at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]

        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]

        at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]

        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:290) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:73) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:154) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:90) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.node.Node.start(Node.java:696) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.5.1.jar:7.5.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.1.jar:7.5.1]

        ... 6 more

[2020-05-29T12:58:55,584][INFO ][o.e.n.Node               ] [wazuh-node-1] stopping ...

[2020-05-29T12:58:55,643][INFO ][o.e.n.Node               ] [wazuh-node-1] stopped

[2020-05-29T12:58:55,644][INFO ][o.e.n.Node               ] [wazuh-node-1] closing ...

[2020-05-29T12:58:55,686][INFO ][o.e.n.Node               ] [wazuh-node-1] closed

[2020-05-29T12:58:55,694][INFO ][o.e.x.m.p.NativeController] [wazuh-node-1] Native controller process has stopped - no new native processes can be started

[Alberto Rodriguez]

Hello SS
Sorry for the late response. I recommend you open a new thread always help with a non-related issue to this thread.

[2020-05-29T12:58:55,551][ERROR][o.e.g.GatewayMetaState   ] [wazuh-node-1] failed to read or upgrade local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /home/wazuh3/elasticsearch-7.5.1/data/nodes/0/_state/manifest-12290.st

This message usually means that there is a reading file problem due to wrong permissions, no data in the mentioned folder, corrupted data… etc. Could you please verify that your /home/wazuh3/elasticsearch-7.5.1/ folder is correct? Can you please share your elasticsearch.yml configuration file?

Best regards,
Alberto R