Issue running indexer-security-init.sh before cluster upgrade

[felixm]

I installed a Wazuh 4.12 cluster envrionment using the ansible deployement method several months ago everything appears to be running properly.  I would like to upgrade to 4.14 and in following the documentation I'm running the /usr/share/wazuh-indexer/bin/indexer-security-init.sh to back up the security configuration and I receive a runtime exception error.

/usr/share/wazuh-indexer/bin/indexer-security-init.sh --options "-backup /etc/wazuh-indexer/opensearch-security -icl -nhnv"

wzinds01:~# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Security Admin v7
Will connect to 10.X.X.2:9300 ... done
ERR: An unexpected RuntimeException occured: error while performing request
Trace:
java.lang.RuntimeException: error while performing request
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:1257)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:358)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:575)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:165)
Caused by: org.apache.http.ProtocolException: Not a valid protocol version: This is not an HTTP port
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:209)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:245)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.http.ParseException: Not a valid protocol version: This is not an HTTP port
        at org.apache.http.message.BasicLineParser.parseProtocolVersion(BasicLineParser.java:148)
        at org.apache.http.message.BasicLineParser.parseStatusLine(BasicLineParser.java:366)
        at org.apache.http.impl.nio.codecs.DefaultHttpResponseParser.createMessage(DefaultHttpResponseParser.java:112)
        at org.apache.http.impl.nio.codecs.DefaultHttpResponseParser.createMessage(DefaultHttpResponseParser.java:50)
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.parseHeadLine(AbstractMessageParser.java:156)
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:207)
        ... 11 more

According to GET _cluster/health/ the cluster is green
{
  "cluster_name": "wazuh",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 2,
  "number_of_data_nodes": 2,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 559,
  "active_shards": 1073,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

wzinds01:~# curl -k -u admin https://10.X.X.2:9200/_cat/nodes?v
Enter host password for user 'admin':
ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles                                        cluster_manager name
10.X.X.3           85          97   6    0.28    0.58     0.60 dimr      cluster_manager,data,ingest,remote_cluster_client *               wzinds02-i
10.X.X.2           87          98   8    0.72    0.50     0.61 dimr      cluster_manager,data,ingest,remote_cluster_client -               wzinds01-i

filebeat tests ok, so I don't believe it is certificate related

wzinds01:~# filebeat test output
elasticsearch: https://10.X.X.2:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.X.X.2
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
elasticsearch: https://10.X.X.3:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.X.X.3
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2

journalctl -xe -u wazuh-indexer is listing some warnings:

Nov 04 14:54:24 wzinds01 systemd-entrypoint[2503054]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 04 14:54:24 wzinds01 systemd-entrypoint[2503054]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.19.1.jar)
Nov 04 14:54:24 wzinds01 systemd-entrypoint[2503054]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 04 14:54:24 wzinds01 systemd-entrypoint[2503054]: WARNING: System::setSecurityManager will be removed in a future release
Nov 04 14:54:25 wzinds01 systemd-entrypoint[2503054]: Nov 04, 2025 2:54:25 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Nov 04 14:54:25 wzinds01 systemd-entrypoint[2503054]: WARNING: COMPAT locale provider will be removed in a future release
Nov 04 14:54:26 wzinds01 systemd-entrypoint[2503054]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 04 14:54:26 wzinds01 systemd-entrypoint[2503054]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.19.1.jar)
Nov 04 14:54:26 wzinds01 systemd-entrypoint[2503054]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 04 14:54:26 wzinds01 systemd-entrypoint[2503054]: WARNING: System::setSecurityManager will be removed in a future release
Nov 04 14:54:37 wzinds01 systemd[1]: Started wazuh-indexer.service - wazuh-indexer.
Subject: A start job for unit wazuh-indexer.service has finished successfully
Defined-By: systemd
Support: http://www.ubuntu.com/support
A start job for unit wazuh-indexer.service has finished successfully.

The job identifier is 4537680

journalctl -u wazuh-indexer.service | grep -i -E "error"

Nov 01 00:00:01 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 01 00:00:01 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 02 00:00:01 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 02 00:00:01 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 03 00:00:01 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 03 00:00:01 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 04 00:00:00 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Nov 04 00:00:00 wzinds01 systemd-entrypoint[2385277]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")

 

Please let me know if there is anything else I can share to troubleshoot.

Thanks,

Felix

[juanjos...@wazuh.com]

Hi Felix, I'll be working with you to solve this issue, just let me do some research and I will back to you shortly

[juanjos...@wazuh.com]

Hi Felix, sorry for the delay, reading your issue it seems to be related with some port problem.

From your output your port is trying to connect to 10.X.X.2:9300, but port 9300 is for inter-node communication, not for the security admin tool.

Please try:

export JAVA_HOME=/usr/share/wazuh-indexer/jdk
/usr/share/wazuh-indexer/bin/indexer-security-init.sh --options "-h 10.X.X.2 -p 9200 -backup /etc/wazuh-indexer/opensearch-security -icl -nhnv -cacert /etc/wazuh-indexer/certs/root-ca.pem -cert /etc/wazuh-indexer/certs/admin.pem -key /etc/wazuh-indexer/certs/admin-key.pem"

pointing to the port 9200

[felixm]

Looks like even after specifying the host IP and port number it is still trying to use port 9300

[juanjos...@wazuh.com]

Hi Felix, 


Can you share the output of:

systemctl status indexer
systemctl status manager


also can you share the content of the file:

indexer-security-init.sh



in the meanwhile, you can try this indexer-security-init.sh:

https://github.com/wazuh/wazuh-indexer/blob/main/distribution/src/bin/indexer-security-init.sh

[felixm]

Wazuh Manager service:

Wazuh Index Service:

indexer-security-init.sh

#!/bin/bash

# Wazuh-indexer securityadmin wrapper
# Copyright (C) 2022, Wazuh Inc.
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.

CONFIG_PATH="/etc/wazuh-indexer"

if [ ! -d "${CONFIG_PATH}" ]; then
    echo "ERROR: it was not possible to find ${CONFIG_PATH}"
    exit 1
fi

CONFIG_FILE="${CONFIG_PATH}/opensearch.yml"

if [ ! -f "${CONFIG_FILE}" ]; then
    echo "ERROR: it was not possible to find ${CONFIG_FILE}"
    exit 1
fi

INSTALL_PATH="/usr/share/wazuh-indexer"

if [ ! -d "${INSTALL_PATH}" ]; then
        echo "ERROR: it was not possible to find ${INSTALL_PATH}"
        exit 1
fi

HOST=""
OPTIONS="-icl -nhnv"
WAZUH_INDEXER_ROOT_CA="$(cat ${CONFIG_FILE} 2>&1 | grep http.pemtrustedcas | sed 's/.*: //' | tr -d "[\"\']")"
WAZUH_INDEXER_ADMIN_PATH="$(dirname "${WAZUH_INDEXER_ROOT_CA}" 2>&1)"
SECURITY_PATH="${INSTALL_PATH}/plugins/opensearch-security"
SECURITY_CONFIG_PATH="${CONFIG_PATH}/opensearch-security"

# -----------------------------------------------------------------------------

trap ctrl_c INT

clean(){

    exit_code=$1
    indexer_process_id=$(pgrep -f wazuh-indexer -c)
    if [ "${indexer_process_id}" -gt 1 ]; then
        pkill -n -f wazuh-indexer
    fi
    exit "${exit_code}"

}

ctrl_c() {
    clean 1
}

# -----------------------------------------------------------------------------

getNetworkHost() {

    HOST=$(grep -hr "network.host:" "${CONFIG_FILE}" 2>&1)
    NH="network.host: "
    HOST="${HOST//$NH}"
    HOST=$(echo "${HOST}" | tr -d "[\"\']")

    isIP=$(echo "${HOST}" | grep -P "^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$")
    isDNS=$(echo "${HOST}" | grep -P "^[a-zA-Z0-9][a-zA-Z0-9-]{1,61}[a-zA-Z0-9](?:\.[a-zA-Z]{2,})+$")

    # Allow to find ip with an interface
    if [ -z "${isIP}" ] && [ -z "${isDNS}" ]; then
        interface="${HOST//_}"
        HOST=$(ip -o -4 addr list "${interface}" | awk '{print $4}' | cut -d/ -f1)
    fi

    if [ "${HOST}" = "0.0.0.0" ]; then
        HOST="127.0.0.1"
    fi

    if [ -z "${HOST}" ]; then
        echo "ERROR: network host not valid, check ${CONFIG_FILE}"
        exit 1
    fi

}

# -----------------------------------------------------------------------------
getPort() {

    PORT=$(grep -hr 'transport.tcp.port' "${CONFIG_FILE}" 2>&1)
    if [ "${PORT}" ]; then
        PORT=$(echo "${PORT}" | cut -d' ' -f2 | cut -d'-' -f1)
    else
        PORT="9200"
    fi
    PORT=$(echo "${PORT}" | tr -d "[\"\']")

}
# -----------------------------------------------------------------------------

securityadmin() {

    if [ ! -d "${SECURITY_PATH}" ]; then
        echo "ERROR: it was not possible to find ${SECURITY_PATH}"
        exit 1
    elif [ ! -d "${INSTALL_PATH}/jdk" ]; then
        echo "ERROR: it was not possible to find ${INSTALL_PATH}/jdk"
        exit 1
    fi

    if [ -f "${WAZUH_INDEXER_ADMIN_PATH}/admin.pem" ] && [ -f "${WAZUH_INDEXER_ADMIN_PATH}/admin-key.pem" ] && [ -f "${WAZUH_INDEXER_ROOT_CA}" ]; then
        OPENSEARCH_CONF_DIR="${CONFIG_PATH}" JAVA_HOME="${INSTALL_PATH}/jdk" runuser wazuh-indexer --shell="/bin/bash" --command="${SECURITY_PATH}/tools/securityadmin.sh -cd ${SECURITY_CONFIG_PATH} -cacert ${WAZUH_INDEXER_ROOT_CA} -cert ${WAZUH_INDEXER_ADMIN_PATH}/admin.pem -key ${WAZUH_INDEXER_ADMIN_PATH}/admin-key.pem -h ${HOST} -p ${PORT} ${OPTIONS}"
    else
        echo "ERROR: this tool try to find admin.pem and admin-key.pem in ${WAZUH_INDEXER_ADMIN_PATH} but it couldn't. In this case, you must run manually the Indexer security initializer by running the command: JAVA_HOME="/usr/share/wazuh-indexer/jdk" runuser wazuh-indexer --shell="/bin/bash" --command="/usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /etc/wazuh-indexer/opensearch-security -cacert /path/to/root-ca.pem -cert /path/to/admin.pem -key /path/to/admin-key.pem -h ${HOST} -p ${PORT} ${OPTIONS}" replacing /path/to/ by your certificates path."
        exit 1
    fi

}

help() {
    echo
    echo "Usage: $0 [OPTIONS]"
    echo
    echo "    -ho, --host <host>    [Optional] Target IP or DNS to configure security."
    echo "    --port <port>         [Optional] wazuh-indexer security port."
    echo "    --options <options>   [Optional] Custom securityadmin options."
    echo "    -h, --help            Show this help."
    echo
    exit "$1"
}


main() {

    getNetworkHost
    getPort

    while [ -n "$1" ]
    do
        case "$1" in
        "-h"|"--help")
            help 0
            ;;
        "-ho"|"--host")
            if [ -n "$2" ]; then
                HOST="$2"
                HOST=$(echo "${HOST}" | tr -d "[\"\']")
                isIP=$(echo "${2}" | grep -P "^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$")
                isDNS=$(echo "${2}" | grep -P "^[a-zA-Z0-9][a-zA-Z0-9-]{1,61}[a-zA-Z0-9](?:\.[a-zA-Z]{2,})+$")
                if [[ -z "${isIP}" ]] &&  [[ -z "${isDNS}" ]]; then
                    echo "The given information does not match with an IP address or a DNS."
                    exit 1
                fi
                shift 2
            else
                help 1
            fi
            ;;
        "--port")
            if [ -n "$2" ]; then
                PORT="$2"
                PORT=$(echo "${PORT}" | tr -d "[\"\']")
                if [[ -z $(echo "${2}" | grep -P "^([1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$") ]]; then
                    echo "The given information does not match with a valid PORT number."
                    exit 1
                fi
                shift 2
            else
                help 1
            fi
            ;;
        "--options")
            if [ -n "$2" ]; then
                OPTIONS="$2"
                shift 2
            else
                help 1
            fi
            ;;
        *)
            help 1
        esac
    done

    securityadmin

}

main "$@"

[felixm]

Juanjose,

I also tried pulling down a copy of the indexer-security-init.sh, and it is still using the same port.

[juanjos...@wazuh.com]

Hi Felix, 

I need to see more files just to be sure your port 9200 is well configured.


Can you share the content of the file: securityadmin.sh

that is located here:


/usr/share/wazuh-indexer/plugins/opensearch-security/tools/


also could you run this indexer-security-init.sh:

    # Print the port requested
    echo "Port used: ${PORT}"

}
# -----------------------------------------------------------------------------

securityadmin() {

    if [ ! -d "${SECURITY_PATH}" ]; then
        echo "ERROR: it was not possible to find ${SECURITY_PATH}"
        exit 1
    elif [ ! -d "${INSTALL_PATH}/jdk" ]; then
        echo "ERROR: it was not possible to find ${INSTALL_PATH}/jdk"
        exit 1
    fi

    if [ -f "${WAZUH_INDEXER_ADMIN_PATH}/admin.pem" ] && [ -f "${WAZUH_INDEXER_ADMIN_PATH}/admin-key.pem" ] && [ -f "${WAZUH_INDEXER_ROOT_CA}" ]; then

        OPENSEARCH_CONF_DIR="${CONFIG_PATH}" JAVA_HOME="${INSTALL_PATH}/jdk" runuser wazuh-indexer --shell="/bin/bash" --command="${SECURITY_PATH}/tools/securityadmin.sh -cd ${SECURITY_CONFIG_PATH} -c[...]
    else
        echo "ERROR: this tool try to find admin.pem and admin-key.pem in ${WAZUH_INDEXER_ADMIN_PATH} but it couldn't. In this case, you must run manually the Indexer security initializer by running t[...]

Is the same indexer-security-init.sh but just prints the port that is using.

[felixm]

Juanjose,

contents of  securityadmin.sh:
#!/bin/bash

SCRIPT_PATH="${BASH_SOURCE[0]}"
if ! [ -x "$(command -v realpath)" ]; then
    if [ -L "$SCRIPT_PATH" ]; then

        [ -x "$(command -v readlink)" ] || { echo "Not able to resolve symlink. Install realpath or readlink.";exit 1; }

        # try readlink (-f not needed because we know its a symlink)
        DIR="$( cd "$( dirname $(readlink "$SCRIPT_PATH") )" && pwd -P)"
    else
        DIR="$( cd "$( dirname "$SCRIPT_PATH" )" && pwd -P)"
    fi
else
    DIR="$( cd "$( dirname "$(realpath "$SCRIPT_PATH")" )" && pwd -P)"
fi

BIN_PATH="java"

# now set the path to java: first OPENSEARCH_JAVA_HOME, then JAVA_HOME
if [ ! -z "$OPENSEARCH_JAVA_HOME" ]; then
    BIN_PATH="$OPENSEARCH_JAVA_HOME/bin/java"
elif [ ! -z "$JAVA_HOME" ]; then
    BIN_PATH="$JAVA_HOME/bin/java"
else
    echo "WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use $(which $BIN_PATH)"
fi

"$BIN_PATH" $JAVA_OPTS -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=OFF -cp "$DIR/../*:$DIR/../../../lib/*:$DIR/../deps/*" org.opensearch.security.tools.SecurityAdmin "$@" 2>/dev/null

indexer-security-init.sh, port is still the same

[juanjos...@wazuh.com]

Hi Felix, hope this can fix your issue. Please do the following

run this on your terminal:

grep -hr "transport.tcp.port" /etc/wazuh-indexer/opensearch.yml

this will give you the current port in use, please the output of the port, if you see 9300, it means that in your opensearch.yml file, that port is configured for usage. If that is teh case change it to 9200 and try again to run your ininitial:

/usr/share/wazuh-indexer/bin/indexer-security-init.sh --options "-backup /etc/wazuh-indexer/opensearch-security -icl -nhnv"

Note: also by running this could work:


/usr/share/wazuh-indexer/bin/indexer-security-init.sh --options "-backup /etc/wazuh-indexer/opensearch-security -icl -nhnv" --port 9200


we must expect that adding the --port 9200 uses the right one.


My suggestion is, do what I suggest first, if that fix your issue were are done here!!! :D 

[felixm]

Juanjose,

Thank you so much, I saw "transport.tcp.port: 9300-9399" in the opensearch.yml when I started trouble shooting the issue.  I not sure how it got there but I tried commenting it out with a "#" and didnot remove the line completely.  Once I remove the line completely the backup command ran as ecpected.

Thank you again!!!!!!!

[juanjos...@wazuh.com]

No worries!!! I am happy to help! 

enjoy Wazuh!