Elasticsearch could not be started. (Unattended installation)

[joh nte]

Hi,

after trying in vain to update my Wazuh 3.13 to 4.0 (link), i've decided to do a clean installation on a Centos 7 machine.

I've tried the "step by step" installation, but elastic's service didn't go up.. so, after removing all the leftovers, i've tried the Unattended installation, but, after the system verify, the wazuh installation and the creations of the certificates, the installation stop with this error:  
--------------------------------------------------------------------------------------------------------------------------------------------
Certificates created

sed: can't read /etc/elasticsearch/jvm.options: No such file or directory

sed: can't read /etc/elasticsearch/jvm.options: No such file or directory

ln: target ‘/usr/share/elasticsearch/lib/’ is not a directory: No such file or directory

Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Elasticsearch could not be started.

---------------------------------------------------------------------------------------------------------------------------------------------

I didn't skip the healtcheck, and i've saw this log by using the -d flag at the end of the command to download and run the script.

Can someone help me with this?

[David Fernández Miranda]

Hello Joh,

The jvm.options file is embedded in the Elasticsearch package. The /etc/elasticsearch/ directory should have the following content and permissions:

drwxr-sr-x. 2 root elasticsearch   271 Dec  4 15:37 certs

-rw-rw----. 1 root elasticsearch   199 Dec  4 12:01 elasticsearch.keystore

-rw-rw----. 1 root elasticsearch  1399 Dec  4 15:33 elasticsearch.yml

-rw-rw----. 1 root elasticsearch  2581 Dec  4 15:34 jvm.options

drwxr-s---. 2 root elasticsearch     6 Sep  1 21:31 jvm.options.d

-rw-rw----. 1 root elasticsearch 10774 Sep  1 21:28 log4j2.properties

Could you check that this file exists? In case of having this file, which permissions does it have?

Regards,

David

[joh nte]

Hi David.
Thanks for the reply.
In /etc/elasticsearch/ directory i see only one file and one folder!

drwxr-x---. 2 root root 4096 Dec  4 12:52 certs

-rw-r-----. 1 root root 1435 Dec  4 12:52 elasticsearch.yml

what can i do?

[David Fernández Miranda]

Hello Joh, to clean the previous installation you should remove all the remaining directories. You should remove /var/lib/elasticsearch and /etc/elasticsearch/. Then, to make sure that you uninstall all Elasticsearch dependencies, run the following command:

yum autoremove opendistro* -y

Then you'll be able to reinstall Elasticsearch.

Regards,

David

[joh nte]

Thanks David.

I've done what you suggest but the problem still remain, the installation quits at this point:

Created 4 node certificates.

Created 2 client certificates.

Certificates created

Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Elasticsearch could not be started.

here's the elastichsearch's log:

[2020-12-10T11:23:02,136][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: org.elasticsearch.bootstrap.BootstrapException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/lib/tools.jar

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.1.jar:7.9.1]

        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]

Caused by: org.elasticsearch.bootstrap.BootstrapException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/lib/tools.jar

        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]

        ... 6 more

Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/lib/tools.jar

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]

        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]

        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]

        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]

        at java.nio.file.Files.readAttributes(Files.java:1843) ~[?:?]

        at java.util.zip.ZipFile$Source.get(ZipFile.java:1172) ~[?:?]

        at java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:718) ~[?:?]

        at java.util.zip.ZipFile.<init>(ZipFile.java:238) ~[?:?]

        at java.util.zip.ZipFile.<init>(ZipFile.java:168) ~[?:?]

        at java.util.jar.JarFile.<init>(JarFile.java:347) ~[?:?]

        at java.util.jar.JarFile.<init>(JarFile.java:318) ~[?:?]

        at java.util.jar.JarFile.<init>(JarFile.java:257) ~[?:?]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:183) ~[elasticsearch-core-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]

I even tried to remove everithing using the uninstallation guide.. but nothing..

I'm using Centos 7

[David Fernández Miranda]

Hello Joh,

I've replicated your scenario and, as it happened to you, if you reinstall the package, but there were leftovers from the previous installation, the package doesn't create the missing file.

To be able to reinstall elasticsearch I did the following:

• yum autoremove opendistro* filebeat wazuh-manager -y
• rm -rf /etc/elasticsearch/
• rm -rf  /var/lib/elasticsearch/
• rm -rf  /usr/share/elasticsearch/
• rm -rf ~/searchguard/
• rm -rf /etc/kibana/
• rm -rf  /var/lib/kibana/
• rm -rf  /usr/share/kibana/
• rm -rf /etc/filebeat/
• rm -rf  /var/lib/filebeat/
• rm -rf  /usr/share/filebeat/

After this, I ran the script again and had no issues with the installation.

Hope this works for you. If you have any doubt, do not hesitate to ask.

Regards,

David

[joh nte]

Nothing to do..

I've done all the passages suggested by you and, during the scripts, it stops with "Elasticsearch could not be started".
Here's the Elasticsearch log:

[2020-12-10T15:43:58,236][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]

Is this the right script?
 curl -so ~/all-in-one-installation.sh https://raw.githubusercontent.com/wazuh/wazuh-documentation/4.0/resources/open-distro/unattended-installation/all-in-one-installation.sh && bash ~/all-in-one-installation.sh -d

[David Fernández Miranda]

I'm sorry it didn't work for you Joh. It looks like the package was not removed. Could you run the following command to ensure that the Elasticsearch package has been removed?

rpm -qa | grep elasticsearch

Regards,

David 

[joh nte]

Hi David, thank you very much for helping me.
rpm -qa | grep elasticsearch gives no output, so the package has been removed!
Now I'm uploading a clean Centos installation, so i can try installing Wazuh where theoretically there should be no leftover problems!

  
I'll update you, thanks again! 

[David Fernández Miranda]

Hi Joh! 

I'm glad to help you. If you have any other doubt, do not hesitate to ask.

Regards,

David