Dear John,
Thank you for using Wazuh and sorry for the late reply.
Dear John,
Thank you for reaching Wazuh and reaching out to us.
Kindly find summary of steps below for backup and restore of wazuh files to a new server:
BACKUPS:Wazuh manager server:- Create the destination folder to store the files. For version control, add the date and time of the backup to the name of the folder:
bkp_folder=~/wazuh_files_backup/$(date +%F_%H:%M)
mkdir -p $bkp_folder && echo $bkp_folder- Save the host information:
cat /etc/*release* > $bkp_folder/host-info.txt
echo -e "\n$(hostname): $(hostname -I)" >> $bkp_folder/host-info.txt- Back up the Wazuh server data and configuration files as described below:
rsync -aREz \
/etc/filebeat/ \
/etc/postfix/ \
/var/ossec/api/configuration/ \
/var/ossec/etc/client.keys \
/var/ossec/etc/sslmanager* \
/var/ossec/etc/ossec.conf \
/var/ossec/etc/internal_options.conf \
/var/ossec/etc/local_internal_options.conf \
/var/ossec/etc/rules/local_rules.xml \
/var/ossec/etc/decoders/local_decoder.xml \
/var/ossec/etc/shared/ \
/var/ossec/logs/ \
/var/ossec/queue/agentless/ \
/var/ossec/queue/agents-timestamp \
/var/ossec/queue/fts/ \
/var/ossec/queue/rids/ \
/var/ossec/stats/ \
/var/ossec/var/multigroups/ $bkp_folder- If present, back up certificates and additional configuration files:
rsync -aREz \ /var/ossec/etc/*.pem \
/var/ossec/etc/authd.pass $bkp_folder- Backup your custom files. If you have custom active responses, CDB lists, integrations, or wodles, kindly adapt the below commands accordingly:
rsync -aREz \
/var/ossec/active-response/bin/<custom_AR_script> \
/var/ossec/etc/lists/<user_cdb_list>.cdb \
/var/ossec/integrations/<custom_integration_script> \
/var/ossec/wodles/<custom_wodle_script> $bkp_folderReplace
<custom_AR_script> with the name of the AR script,
<user_cdb_list> with the name of the cdb list,
<custom_integration_script> with the name of the integration script and
<custom_wodle_script> with the name if the wodle script.
- Stop the wazuh manager service to prevent modification attempts while copying the Wazuh databases.
systemctl stop wazuh-manager- Back up the Wazuh databases as they hold collected data from the agents.
rsync -aREz \
/var/ossec/queue/db/ $bkp_folder- Start the Wazuh manager service
systemctl start wazuh-managerElasticsearch:- Back up the elasticsearch certificates and configuration files:
rsync -aREz \
/etc/elasticsearch/certs/ \
/etc/elasticsearch/jvm.options \
/etc/elasticsearch/jvm.options.d \
/etc/elasticsearch/log4j2.properties \
/etc/elasticsearch/elasticsearch.yml \
/etc/elasticsearch/elasticsearch.keystore \
/etc/elasticsearch/elasticsearch-plugins.example.yml \
/etc/elasticsearch/role_mapping.yml \
/etc/elasticsearch/roles.yml \
/etc/elasticsearch/users \
/etc/elasticsearch/users_roles \
/usr/lib/sysctl.d/elasticsearch.conf $bkp_folder Kibana:- Back up the Wazuh dashboard certificates and configuration files:
rsync -aREz \
/etc/kibana/certs/ \
/etc/kibana/kibana.yml \
/etc/kibana/kibana.keystore \
/etc/kibana/node.options \
/usr/share/kibana/data/wazuh/config/wazuh.yml \
/usr/share/kibana/data/wazuh/config/wazuh-registry.json $bkp_folder- Additional Kibana files backup (if present):
rsync -aREz \
/usr/share/kibana/data/wazuh/downloads \
/usr/share/kibana/plugins/wazuh/public/assets/custom/images $bkp_folder Check and verify that the Wazuh manager is active and list all the backed up files respectively with the commands below:
systemctl status wazuh-manager
find $bkp_folder -type f | sed "s|$bkp_folder/||" | lessMigrating to a new server.Note: You need to make sure you have a new installation of Wazuh. You can use the
guide to perform a fresh installation of the central components on the new server.
- Compress the files generated after performing Wazuh files backup and transfer them to the new server:
tar -cvzf wazuh_central_components.tar.gz ~/wazuh_files_backup/- Move the compressed file to the root / directory of your node:
mv wazuh_central_components.tar.gz / cd /- Decompress the backup files and change the current working directory to the directory based on the date and time of the backup files:
tar -xzvf wazuh_central_components.tar.gz cd ~/wazuh_files_backup/<DATE_TIME>Restoring Elasticsearch files:- Stop the elasticsearch to prevent any modifications to the elasticsearch files during the restoration process:
systemctl stop elasticsearch- Restore the elasticsearch configuration files and change the file permissions and ownerships accordingly:
sudo cp etc/elasticsearch/jvm.options /etc/elasticsearch/jvm.options
chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options
sudo cp -r etc/elasticsearch/jvm.options.d/* /etc/elasticsearch/jvm.options.d/
chown elasticsearch:elasticsearch /etc/elasticsearch/jvm.options.d
sudo cp etc/elasticsearch/log4j2.properties /etc/elasticsearch/log4j2.properties
chown elasticsearch:elasticsearch /etc/elasticsearch/log4j2.properties
sudo cp etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
chown elasticsearch:elasticsearch /etc/elasticsearch/elasticsearch.yml
sudo cp etc/elasticsearch/elasticsearch.keystore /etc/elasticsearch/elasticsearch.keystore
chown elasticsearch:elasticsearch /etc/elasticsearch/elasticsearch.keystore
sudo cp etc/elasticsearch/elasticsearch-plugins.example.yml /etc/elasticsearch/elasticsearch-plugins.example.yml
chown elasticsearch:elasticsearch /etc/elasticsearch/elasticsearch-plugins.example.yml
sudo cp etc/elasticsearch/elasticsearch-plugins.example.yml /etc/elasticsearch/elasticsearch-plugins.example.yml
chown elasticsearch:elasticsearch /etc/elasticsearch/elasticsearch-plugins.example.yml
sudo cp etc/elasticsearch/role_mapping.yml /etc/elasticsearch/role_mapping.yml
chown elasticsearch:elasticsearch /etc/elasticsearch/role_mapping.yml sudo cp etc/elasticsearch/roles.yml /etc/elasticsearch/roles.yml
chown elasticsearch:elasticsearch /etc/elasticsearch/roles.yml
sudo cp etc/elasticsearch/users /etc/elasticsearch/users
chown elasticsearch:elasticsearch /etc/elasticsearch/users
sudo cp etc/elasticsearch/users_roles /etc/elasticsearch/users_roles
chown elasticsearch:elasticsearch /etc/elasticsearch/users_roles
sudo cp usr/lib/sysctl.d/elasticsearch.conf /usr/lib/sysctl.d/elasticsearch.conf- Start the elasticsearch service:
systemctl start elasticsearchRestoring Wazuh server files:- Stop the Wazuh manager and Filebeat to prevent any modification to the Wazuh server files during the restore process:
systemctl stop filebeat
systemctl stop wazuh-manager- Copy the Wazuh server data and configuration files, and change the file permissions and ownerships accordingly:
sudo cp etc/filebeat/filebeat.reference.yml /etc/filebeat/
sudo cp etc/filebeat/fields.yml /etc/filebeat/
sudo cp -r etc/filebeat/modules.d/* /etc/filebeat/modules.d/
sudo cp -r etc/postfix/* /etc/postfix/
sudo cp var/ossec/etc/client.keys /var/ossec/etc/
chown root:wazuh /var/ossec/etc/client.keys
sudo cp -r var/ossec/etc/sslmanager* /var/ossec/etc/
sudo cp var/ossec/etc/ossec.conf /var/ossec/etc/
chown root:wazuh /var/ossec/etc/ossec.conf
sudo cp var/ossec/etc/internal_options.conf /var/ossec/etc/
chown root:wazuh /var/ossec/etc/internal_options.conf
sudo cp var/ossec/etc/local_internal_options.conf /var/ossec/etc/
chown root:wazuh /var/ossec/etc/local_internal_options.conf
sudo cp -r var/ossec/etc/rules/* /var/ossec/etc/rules/
chown -R wazuh:wazuh /var/ossec/etc/rules/
sudo cp -r var/ossec/etc/decoders/* /var/ossec/etc/decoders chown -R wazuh:wazuh /var/ossec/etc/decoders/
sudo cp -r var/ossec/etc/shared/* /var/ossec/etc/shared/
chown -R wazuh:wazuh /var/ossec/etc/shared/
chown root:wazuh /var/ossec/etc/shared/ar.conf
sudo cp -r var/ossec/logs/* /var/ossec/logs/
chown -R wazuh:wazuh /var/ossec/logs/
sudo cp -r var/ossec/queue/agentless/* /var/ossec/queue/agentless/
chown -R wazuh:wazuh /var/ossec/queue/agentless/
sudo cp var/ossec/queue/agents-timestamp /var/ossec/queue/
chown root:wazuh /var/ossec/queue/agents-timestamp
sudo cp -r var/ossec/queue/fts/* /var/ossec/queue/fts/
chown -R wazuh:wazuh /var/ossec/queue/fts/
sudo cp -r var/ossec/queue/rids/* /var/ossec/queue/rids/
chown -R wazuh:wazuh /var/ossec/queue/rids/
sudo cp -r var/ossec/stats/* /var/ossec/stats/ chown -R wazuh:wazuh /var/ossec/stats/
sudo cp -r var/ossec/var/multigroups/* /var/ossec/var/multigroups/
chown -R wazuh:wazuh /var/ossec/var/multigroups/
- Restore certificates for Wazuh agent and Wazuh server communication, and additional configuration files if present:
sudo cp -r var/ossec/etc/*.pem /var/ossec/etc/
chown -R root:wazuh /var/ossec/etc/*.pem
sudo cp var/ossec/etc/authd.pass /var/ossec/etc/
chown -R root:wazuh /var/ossec/etc/authd.pass- Restore your custom files. If you have custom active response scripts, CDB lists, integrations, or wodles, adapt the following commands accordingly:
sudo cp var/ossec/active-response/bin/<CUSTOM_ACTIVE_RESPONSE_SCRIPT> /var/ossec/active-response/bin/
chown root:wazuh /var/ossec/active-response/bin/<CUSTOM_ACTIVE_RESPONSE_SCRIPT>
sudo cp var/ossec/etc/lists/<USER_CDB_LIST>.cdb /var/ossec/etc/lists/
chown root:wazuh /var/ossec/etc/lists/<USER_CDB_LIST>.cdb
sudo cp var/ossec/integrations/<CUSTOM_INTEGRATION_SCRIPT> /var/ossec/integrations/
chown root:wazuh /var/ossec/integrations/<CUSTOM_INTEGRATION_SCRIPT>
sudo cp var/ossec/wodles/<CUSTOM_WODLE_SCRIPT> /var/ossec/wodles/
chown root:wazuh /var/ossec/wodles/<CUSTOM_WODLE_SCRIPT>- Restore the Wazuh databases that contain collected data from the Wazuh agents:
sudo cp var/ossec/queue/db/* /var/ossec/queue/db/
chown -R wazuh:wazuh /var/ossec/queue/db/- Start the Wazuh filebeat and Wazuh manager service respectively:
systemctl start filebeat
systemctl start wazuh-managerRestoring Kibana files:Perform the following steps to restore Wazuh reports and custom images on the new server if you have any from your backup.
- Restore your Wazuh reports using the following command:
mkdir -p /usr/share/kibana/data/wazuh/downloads/reports/
sudo cp -r usr/share/kibana/data/wazuh/downloads/reports/* /usr/share/kibana/data/wazuh/downloads/reports/
chown -R kibana:kibana /usr/share/kibana/data/wazuh/downloads/Navigate to
Settings > Configuration > Custom branding from the Kibana dashboard and upload your custom images
Restoring old logs:Wazuh, by default, compresses logs that are older than a day. While performing old log restoration in the
Restoring Wazuh server files section, the old logs remain compressed.
Perform the following actions on your Wazuh server to decompress these logs and index them in the new Elasticsearch:
- Create a Python script called recovery.py on your Wazuh server. This script decompresses all the old logs and stores them in the recovery.py file in the /tmp directory:
touch recovery.py- Add the following content to the recovery.py script:
#!/usr/bin/env python
import gzip
import time
import json
import argparse
import re
import os
from datetime import datetime
from datetime import timedelta
def log(msg):
now_date = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
final_msg = "{0} wazuh-reinjection: {1}".format(now_date, msg)
print(final_msg)
if log_file:
f_log.write(final_msg + "\n")EPS_MAX = 400
wazuh_path = '/var/ossec/'
max_size=1
log_file = None
parser = argparse.ArgumentParser(description='Reinjection script')
parser.add_argument('-eps','--eps', metavar='eps', type=int, required = False, help='Events per second.')
parser.add_argument('-min', '--min_timestamp', metavar='min_timestamp', type=str, required = True, help='Min timestamp. Example: 2017-12-13T23:59:06')
parser.add_argument('-max', '--max_timestamp', metavar='max_timestamp', type=str, required = True, help='Max timestamp. Example: 2017-12-13T23:59:06')
parser.add_argument('-o', '--output_file', metavar='output_file', type=str, required = True, help='Output filename.')
parser.add_argument('-log', '--log_file', metavar='log_file', type=str, required = False, help='Logs output')
parser.add_argument('-w', '--wazuh_path', metavar='wazuh_path', type=str, required = False, help='Path to Wazuh. By default:/var/ossec/')
parser.add_argument('-sz', '--max_size', metavar='max_size', type=float, required = False, help='Max output file size in Gb. Default: 1Gb. Example: 2.5')
args = parser.parse_args()if args.log_file:
log_file = args.log_file
f_log = open(log_file, 'a+')
if args.max_size:
max_size = args.max_size
if args.wazuh_path:
wazuh_path = args.wazuh_path
output_file = args.output_file
#Gb to bytes
max_bytes = int(max_size * 1024 * 1024 * 1024)
if (max_bytes <= 0):
log("Error: Incorrect max_size") exit(1)
month_dict = ['Null','Jan','Feb','Mar','Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov','Dec']
if args.eps:
EPS_MAX = args.eps
if EPS_MAX < 0:
log("Error: incorrect EPS")
exit(1)
min_date = re.search('(\\d\\d\\d\\d)-(\\d\\d)-(\\d\\d)T\\d\\d:\\d\\d:\\d\\d', args.min_timestamp)
if min_date:
min_year = int(min_date.group(1))
min_month = int(min_date.group(2))
min_day = int(min_date.group(3))else:
log("Error: Incorrect min timestamp")
exit(1)
max_date = re.search('(\\d\\d\\d\\d)-(\\d\\d)-(\\d\\d)T\\d\\d:\\d\\d:\\d\\d', args.max_timestamp)
if max_date:
max_year = int(max_date.group(1))
max_month = int(max_date.group(2))
max_day = int(max_date.group(3))
else:
log("Error: Incorrect max timestamp")
exit(1)
# Converting timestamp args to datetime
min_timestamp = datetime.strptime(args.min_timestamp, '%Y-%m-%dT%H:%M:%S')
max_timestamp = datetime.strptime(args.max_timestamp, '%Y-%m-%dT%H:%M:%S')
chunk = 0written_alerts = 0
trimmed_alerts = open(output_file, 'w')
max_time=datetime(max_year, max_month, max_day)
current_time=datetime(min_year, min_month, min_day)
while current_time <= max_time:
alert_file = "{0}logs/alerts/{1}/{2}/ossec-alerts-{3:02}.json.gz".format(wazuh_path,current_time.year,month_dict[current_time.month],current_time.day)
if os.path.exists(alert_file):
daily_alerts = 0
compressed_alerts = gzip.open(alert_file, 'r')
log("Reading file: "+ alert_file)
for line in compressed_alerts:
# Transform line to json object
try:
line_json = json.loads(line.decode("utf-8", "replace")) # Remove unnecessary part of the timestamp
string_timestamp = line_json['timestamp'][:19]
# Ensure timestamp integrity
while len(line_json['timestamp'].split("+")[0]) < 23:
line_json['timestamp'] = line_json['timestamp'][:20] + "0" + line_json['timestamp'][20:]
# Get the timestamp readable
event_date = datetime.strptime(string_timestamp, '%Y-%m-%dT%H:%M:%S')
# Check the timestamp belongs to the selected range
if (event_date <= max_timestamp and event_date >= min_timestamp):
chunk+=1 trimmed_alerts.write(json.dumps(line_json))
trimmed_alerts.write("\n")
trimmed_alerts.flush()
daily_alerts += 1
if chunk >= EPS_MAX:
chunk = 0
time.sleep(2)
if os.path.getsize(output_file) >= max_bytes:
trimmed_alerts.close()
log("Output file reached max size, setting it to zero and restarting")
time.sleep(EPS_MAX/100)
trimmed_alerts = open(output_file, 'w')
except ValueError as e:
print("Oops! Something went wrong reading: {}".format(line))
print("This is the error: {}".format(str(e))) compressed_alerts.close()
log("Extracted {0} alerts from day {1}-{2}-{3}".format(daily_alerts,current_time.day,month_dict[current_time.month],current_time.year))
else:
log("Couldn't find file {}".format(alert_file))
#Move to next file
current_time += timedelta(days=1)
trimmed_alerts.close()
- Run the command below to make the recovery.py script executable:
chmod +x recovery.py- Execute the script using nohup command in the background to keep it running after the session is closed. It may take time depending on the size of the old logs.
Usage example:
nohup ./recovery.py -eps 500 -min 2023-06-10T00:00:00 -max 2023-06-18T23:59:59 -o /tmp/recovery.json -log ./recovery.log -sz 2.5 &- Add the /tmp/recovery.json path to the Wazuh Filebeat module /usr/share/filebeat/module/wazuh/alerts/manifest.yml so that Filebeat sends the old alerts to the elasticsearch for indexing:
module_version: 0.1
var:
- name: paths
default:
- /var/ossec/logs/alerts/alerts.json
- /tmp/recovery.json
- name: index_prefix
default: wazuh-alerts-4.x-
input: config/alerts.yml
ingest_pipeline: ingest/pipeline.json
- Restart Filebeat for the changes to take effect:
systemctl restart filebeatVerifying data restoration:
Using the Kibana dashboard, navigate to the Security events, Integrity monitoring, Vulnerabilities, and any other modules to see if the data is restored successfully.
I'm also attaching some reference links below which could be helpful:
https://documentation.wazuh.com/current/user-manual/files-backup/restoring/wazuh-central-components.html#single-node-data-restorationhttps://documentation.wazuh.com/current/user-manual/files-backup/creating/wazuh-central-components.htmlI hope this was helpful. Do not hesitate to get back to us on the outcome or if you have any other query.
Best regards.