Locked Out

[Geoff Getty]

Hello - 

I am a new administrator at an an organization where my predecessor had setup a Wazah system.  As far as I am aware, the system is running and agents are collecting data.

Unfortunately, credentials were not documented accurately, and I have not been able to access the system. The former admin tole me the "Open Distro for Elasticsearch / Kibana" login was tied to Active Directory credentials.  However, none of our Active Directory accounts (including Domain Admins) can log into the prompt.

The system appears to be running on Ubuntu, and I can SSH in using my AD Domain Admin credentials.  However these credentials do not appear to have root access.  The root credentials the former admin left do not work and they do not remember any other credentials.

I know nothing about Wazah / Kibana / Elastic Stack and very little about Linux.  In this situation, is there anything I can do to gain access to the system, or are we looking at having to completely re-make it?

I appreciate any assistance, thank you.

[Dario Menten]

Hello Geoff,
Regarding changing the terminal as root, you need to reset the root password and you will have access to the terminal with all the permissions. To do that, you will have to make use of any of the tutorials you can find over the internet regarding how to reset the root password.
Once you get access as root, you can reset the admin password of OpenDistro, and get access to your Kibana Interface.
You can change the admin password by following this Wazuh Documentation article: Wazuh documentation - Change users’ password
Having in mind this is an Opendistro feature, you should check also the OpenDistro documentation: Apply configuration changes using securityadmin.sh
I will explain to you how to do it easily:

1- Create a new password hash

Passwords are stored in a hash format in a file named internal_users.yml file. As the name suggests this file stores user credentials including the admin user. Therefore, the first step is to create the hash. Elasticsearch provides a script to create a password hash

export JAVA_HOME=/usr/share/elasticsearch/jdk/ &&  /usr/share/elasticsearch/plugins/opendistro_security/tools/hash.sh

This will prompt you to enter a password and outputs its hash. Copy the output as we are going to store it in internal_users.yml

2 - Update internal_users.yml

vi /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml

Find the “admin” user section and update the hash value:

admin:
  hash: "$2y$12$UrLh1/OG6EhZ8RC27BD5ROOQ.4ioECP7RbE1ILC9vg4bWo0cx6vhS"
  reserved: true
  backend_roles:
  - "admin"
  description: "Demo admin user"

3 - Apply security changesThe opendistro_security plugin stores users and permissions in an index (.opendistro_security) and therefore we need to update the index after making any changes to opendistro_security configurations. This is done through securityadmin.sh script which is under opendistro_security tools.

Simply run the following commands:

export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem -h <elasticsearch_IP>

Output:

Open Distro Security Admin v7
Will connect to localhost:9300 ... done
...
...
...
Done with success

You should see “Done with success” at the bottom of the output.
I hope this information could be helpful.

​

[Geoff Getty]

Thank you for the reply and detailed instructions.

Unfortunately, it appears I can't change the root password because the only account I have (AD Domain Admin user) is "not in the sudoers file".

I've tried editing the sudoers file but of course can't do that without sudo privileges as well.

Anything else you can think of that would work?

[Dario Menten]

Hello Geoff,

The first link I sent you, it is explained how to do it in boot recovery mode. In the case you do not have ubuntu, you can boot with another Linux live CD and do the same steps.

I hope this can be helpful.

I look forward to your feedback

[Geoff Getty]

Hello - 

Thank you for the response.  I was able to reset the root password and gain access. 

I was able to generate the password hash for 

I have gotten to the last line of your commands and am receiving the following error:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Open Distro Security Admin v7

Will connect to 127.0.0.1:9300 ... done
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.amazon.opendistroforelasticsearc                                h.security.OpenDistroSecurityPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:722)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:124)
        at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:114)
        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:139)
        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:282)
        at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin$TransportClientImpl.<init>(OpenDistroSecurityAdmin.java:1007)
        at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:520)
        at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:157)
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:713)
        ... 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: ElasticsearchException[Unable to read /etc/elasticsearch/certs/admin.key (/etc/elasticsearch/certs/admin.key). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemkey_filepath]]; nested: ElasticsearchException[Unable to read /etc/elasticsearch/certs/admin.key (/etc/elasticsearch/certs/admin.key). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemkey_filepath];
        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:409)
        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:248)
        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.<init>(DefaultOpenDistroSecurityKeyStore.java:169)
        at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.<init>(OpenDistroSecurity SSLPlugin.java:217)
        at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.<init>(OpenDistroSecurityPlugin.java:246)
        ... 12 more
Caused by: ElasticsearchException[Unable to read /etc/elasticsearch/certs/admin.key (/etc/elasticsearch/certs/admin.key). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemkey_filepath]
        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:948)
        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:227)
        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:387)
        ... 16 more

Running the command w/ the IP of the actual server running Elastric Search  / Wazuh fails:

WARNING: JAVA_HOME not set, will use /usr/bin/java

Open Distro Security Admin v7

Will connect to 192.168.42.168:9300
ERR: Seems there is no Elasticsearch running on 192.168.42.168:9300 - Will exit

Even though I can get to the OpenDistro for Elastic Search / Kibana login page at that IP address.

I'd appreciate any insight.

[Dario Menten]

Hello Geoff,
As far as I can see this is the error:

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: ElasticsearchException[Unable to read /etc/elasticsearch/certs/admin.key (/etc/elasticsearch/certs/admin.key). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemkey_filepath]]; nested: ElasticsearchException[Unable to read /etc/elasticsearch/certs/admin.key (/etc/elasticsearch/certs/admin.key). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemkey_filepath]

It means the script can not access /etc/elasticsearch/certs/admin.key file, even because it does not exist or the user elasticsearch does not have permission to read it, please check this with the following command:

ls -l /etc/elasticsearch/certs/

If the file is there, check the permissions, and set read permissions to all the users to test:

chmod +r /etc/elasticsearch/certs/*

If the file is not there, you will need to recreate the admin certificate from the root-ca but you will need the private key and the public certificate root-ca.key and root-ca.pem:

Create a private key
openssl genrsa -out admin-key-temp.pem 2048

Convert private key to PKCS#8 format
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin.key

Create a CSR that'll be signed by the CA
openssl req -new -key admin.key -subj "/C=AR/ST=MZA/O=Wazuh/OU=Ops/CN=admin" -out admin.csr

Sign the CSR
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca.key -CAcreateserial -sha256 -out admin.pem -days 3650

If this does not work, try recreating the certificates for all the environment (Filebeat, Elasticsearch, and Kibana): https://documentation.wazuh.com/current/user-manual/certificates.html

I hope this could be helpful.

​

[Geoff Getty]

Thank you very much for the information - looking into this now.  Before I proceed, I wanted to clarify: 

There is no "admin.key" in the  /etc/elasticsearch/certs/ directory. However - I do see an "admin-key.pem". Does this change any of your instructions?

Thank you again.

[Dario Menten]

Hello Geoff,
That file is the same that admin.key but with another name, and the name of the file is not important, you only need the name to use it. Have in mind this, the certificates come in pairs, one is the private key (.key file, or key.pem file), and the other is the public certificate (.pem file).
For the script /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh you need to specify the private key of the admin user under the -key parameter, and the certificate for the admin user under the -cert parameter.
That being said, you can use the admin-key.pem file instead of admin.key and vice-versa.
I hope that answers your question.

​

[Geoff Getty]

Thank you for all the clarification.  I've run the command and it seems like it's at least attempting to execute correctly - however, now I'm stumped.  I'm getting:

Open Distro Security Admin v7

Will connect to (ip address):9300

ERR: Seems there is no Elasticseach running on (ip address):9300 - Will exit

I get this error if I give it the actual IP of the system, or the localhost address.  I am assuming it's the same IP as what the system would report from "ifconfig", which is the same IP I can use in the browser to reach the Open Distro for Elasticsearch login prompt.

Is there a command to see what IP / port Elasticsearch is running on?

[Dario Menten]

Hello Geoff,
Yes, you need to check now first if Elasticsearch is running:

systemctl status elasticsearch

And then see if the ports are open, you can make use of netstat and having in mind the Elasticsearch runs under Java:

netstat -tulnap | grep LISTEN | grep java

The 9200 port is for the API calls, and the 9300 port is for cluster connection.

I hope this information could be helpful.

​

[Geoff Getty]

Thanks again.  Elasticsearch was not running for some reason - I started it using the Systemctl command.

Netstat reports 90644/java listening on both 9200 and 9300

The securityadmin.sh script still fails however.  It won't let me scroll up to see the top of the error messages, but I've attached a screencap of what I could capture.

[Dario Menten]

Hello Geoff,
That is a common issue and it can be skipped by adding the -icl to the securityadmin.sh command as stated in the OpenDistro troubleshooting: Troubleshooting securityadmin.sh

export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem -h <elasticsearch_IP> -icl

There are some other ways of troubleshooting that if is still failing, please check the documentation I send in this message.

I hope this could be helpful

​

[Geoff Getty]

I appreciate all the help you've given.  Ultimately, I wasn't able to overcome the problems with running the  securityadmin.sh script.

I created a new Wazuh server and I'm now joining agents to it.  Windows endpoints that have never run the Wazuh client have installed and registered fine.

However, Windows endpoints that are already running the Wazuh client are not registering, although they are showing up in the new Wazuh server when I run the following command:

Invoke-WebRequest -Uri https://packages.wazuh.com/4.x/windows/wazuh-agent-4.2.6-1.msi -OutFile wazuh-agent-4.2.6.msi; ./wazuh-agent-4.2.6.msi /q WAZUH_MANAGER='(New Wazah Server IP)' WAZUH_REGISTRATION_SERVER=' (New Wazah Server IP)' WAZUH_AGENT_GROUP='Servers'

The agent shows as "never connected" in the Wazuh dashboard.  I feel like I'm missing something easy to prompt the Wazah agent to re-register to the new server.  

Any suggestions?

[Geoff Getty]

Please disregard - I have been able to get agents to register with the new server by fully uninstalling the old Wazuh agent and removing the C:\Program Files (x86)\ossec-agent folder, and reinstalling the new Wazuh agent via the generated Powershell command from the Agents page.  FOr others where this is not an option, I have manually edited the ossec.conf file to point to the new server IP.

Issue Resolved, thanks again!