Issue with Wazuh Kubernetes Deployment

[ak]

I am trying to configure wazuh on Kubernetes and hitting an issue.

I am following the documentation in https://documentation.wazuh.com/current/deployment-options/deploying-with-kubernetes/kubernetes-deployment.html for v4.5.2.

Lab Tests:

Kubernetes is deployed on Ubuntu 22.04.3 LTS with flannel , containerd.

- Below are details for tests for reference:

Below is the cluster:

kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
k8-controller   Ready    control-plane   22m   v1.28.2
k8-node         Ready    <none>          22m   v1.28.2
k8-px2-n1       Ready    <none>          22m   v1.28.2

-Master and the worker nodes have vm.max_map_count configured to 262144.

root@k8-controller:~# sysctl -a | grep -i vm.max_map_count
vm.max_map_count = 262144

root@k8-node:~# sysctl -a | grep -i vm.max_map_count
vm.max_map_count = 262144

root@k8-px2-n1:~# sudo sysctl -a | grep -i vm.max_map_count
vm.max_map_count = 262144

- Followed the documentation to clone the deployment.

git clone https://github.com/wazuh/wazuh-kubernetes.git -b v4.5.2 --depth=1
cd wazuh-kubernetes
wazuh/certs/indexer_cluster/generate_certs.sh
wazuh/certs/dashboard_http/generate_certs.sh
kubectl apply -k envs/local-env/

Output:

$ kubectl apply -k envs/local-env/
namespace/wazuh created
storageclass.storage.k8s.io/wazuh-storage created
configmap/dashboard-conf-tgmhtkc5dm created
configmap/indexer-conf-67g4h64bf2 created
configmap/wazuh-conf-bdfk9f4c4d created
secret/dashboard-certs-85d56b8kbf created
secret/dashboard-cred created
secret/indexer-certs-thhtdtbm5f created
secret/indexer-cred created
secret/wazuh-api-cred created
secret/wazuh-authd-pass created
secret/wazuh-cluster-key created
service/dashboard created
service/indexer created
service/wazuh created
service/wazuh-cluster created
service/wazuh-indexer created
service/wazuh-workers created
deployment.apps/wazuh-dashboard created
statefulset.apps/wazuh-indexer created
statefulset.apps/wazuh-manager-master created
Warning: spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.labelSelector: a null labelSelector results in matching no pod
statefulset.apps/wazuh-manager-worker created

kubectl get sc
NAME            PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
wazuh-storage   microk8s.io/hostpath   Delete          Immediate           false                  5m20s

kubectl get pods -n wazuh
NAME                               READY   STATUS              RESTARTS   AGE
wazuh-dashboard-698fbfc755-ttnth   0/1     ContainerCreating   0          40s
wazuh-indexer-0                    0/1     Pending             0          40s
wazuh-manager-master-0             0/1     Pending             0          40s
wazuh-manager-worker-0             0/1     Pending             0          39s

- When checking logs for indexer, I see the below error:

kubectl logs -n wazuh wazuh-indexer-0
Defaulted container "wazuh-indexer" out of: wazuh-indexer, volume-mount-hack (init), increase-the-vm-max-map-count (init)

- No logs
kubectl logs -n wazuh wazuh-manager-master-0
kubectl logs -n wazuh wazuh-manager-worker-0

# Config files:

cat envs/local-env/kustomization.yml
# Copyright (C) 2019, Wazuh Inc.
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../wazuh
patches:
- path: storage-class.yaml
- path: indexer-resources.yaml
- path: wazuh-resources.yaml

cat envs/local-env/wazuh-resources.yaml
# Copyright (C) 2019, Wazuh Inc.
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: wazuh-manager-worker
  namespace: wazuh
spec:
  replicas: 1


cat envs/local-env/indexer-resources.yaml
# Copyright (C) 2019, Wazuh Inc.
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: wazuh-indexer
  namespace: wazuh
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: wazuh-indexer
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: 1
              memory: 2Gi

cat envs/local-env/storage-class.yaml
# Copyright (C) 2019, Wazuh Inc.
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.

# Wazuh StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: wazuh-storage

# Microk8s is our standard for local development
provisioner: microk8s.io/hostpath

# In case you're running Minikube you can comment the line above and use this one
# provisioner: k8s.io/minikube-hostpath

# If you're using a different provider you can list storage classes
# with: "kubectl get sc" and look for the column "Provisioner"

## Version details:

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

For reference, I am following the below guide:

https://documentation.wazuh.com/current/deployment-options/deploying-with-kubernetes/kubernetes-deployment.html

## Questions:

1. I have set vm.max_map_count = 262144 on the Kubernetes host. When checking logs for wazuh-indexer-0 as example, I see this even after setting it on the host.

kubectl logs -n wazuh wazuh-indexer-0
Defaulted container "wazuh-indexer" out of: wazuh-indexer, volume-mount-hack (init), increase-the-vm-max-map-count (init)

Is there any config file within the wazuh k8 repo that needs to be updated in Kubernetes for this to take affect? If so, it would be great if you show the config file that needs to be updated.

2. Could you please let me know if any other changes are needed for this deployment to work? Do let me know if you need any additional information. I will share them here. 
3. I am using the self signed certificate mentioned in the documentation. Generated using "wazuh/certs/indexer_cluster/generate_certs.sh" and "wazuh/certs/dashboard_http/generate_certs.sh" Are there any other changes needed to the yaml files after this?

4. It would be great if the documentation for Kubernetes deployment could be a bit more detailed if possible.

[Sebastian Dario Bustos]

Hello ak,

Thank you for using Wazuh!!!

The memory on the container seems a little short, I'll advice 4gb  or ram minimum  on the indexer-resources.yml and setting the jvm heap size to 2gb  (these parameters need to be set to the half of the physical memory), you can modify this on the path "wazuh/indexer_stack/wazuh-indexer/cluster/indexer-sts.yaml"  and the settings by default are as follows (lines 67 to 69):

          env:

            - name: OPENSEARCH_JAVA_OPTS

              value: '-Xms1g -Xmx1g -Dlog4j2.formatMsgNoLookups=true'

You should change this to (modify -Xms and -Xmx parameters):

          env:

            - name: OPENSEARCH_JAVA_OPTS

              value: '-Xms2g -Xmx2g -Dlog4j2.formatMsgNoLookups=true'

There shouldn't be more modifications to make for your cluster to work, once the certificates are generated all the cluster should go up without further change.

Let me know.

Regards.