export all real time log from wajuh server to s3 bucket

[maurya poonam]

Hi

   I am facing problem with 1 month and done everything from documentation  still i am unable to export my real time logs like whatever folder or file created or deleted or any changes etc. kindly provide me solution.

Thanks and regards

Poonam Maurya

[maurya poonam]

I want to export real time logs in s3 bucket from wajuh server or wajuh dashboard. Logs are shown on Wajuh dashboard but I am unable to export all log in s3 bucket.

[Antonio Kim (Wazuh)]

Hi Maurya!
Thanks for using Wazuh.

Could you describe me the steps you are using to export the logs to the s3 bucket?

Regards

Antonio

[maurya poonam]

Step-1 Firstly I create a role and attch policy PutObject and ListBucket and AmazonS3FullAccess,AmazonSSMFullAccess.
  step-2 create S3 bucket
  Step-3 Configure Wajuh Manger and Wajuh Agent
  Step-4 Write a script for export logs from wajuh server to S3 bucket.

[Antonio Kim (Wazuh)]

Thank you, Maurya, for the details.

I wanted to ask if you were able to find the logs you want to export within the Wazuh platform. Regarding exporting it via a script, could you please provide more information about that?

Regards

[Antonio Kim (Wazuh)]

Apologies, I'll add one more query to better understand the context of your issue. Would the central components of Wazuh be installed on an EC2 instance?

I'll await your response.

Regards

Antonio

[maurya poonam]

Yes, central components are installed successfully and wajuh dashboard is running successfully and syslog, maillog, secure log audit log, and whatever file and folder created or deleted or any changes every log show on Wajuh server.. But my problem is I want to export all this log on S3 bucket as well as my real time log ex any file folder deleted or created or any changes in directory. Guide me where and what changes we have to do? How can I export all log in S3 bucket in less size. 

Thanks

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/wabOSjj55xM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6b5ddf40-8e83-44e8-8fb7-1ef9bd359e23n%40googlegroups.com.

[Antonio Kim (Wazuh)]

Hello, dear Maurya Poonam.

So, I understand that you have successfully installed the core components of Wazuh (manager, indexer, filebeat, dashboard) on an EC2 instance. What you are looking to do is transfer log and alert information in real time to an S3 bucket.

To begin, it's important for you to know that the log files are located in /var/ossec/logs. Alert logs are stored in /var/ossec/alerts. General logs in ossec.log rotate daily and are stored in /var/ossec/logs/wazuh according to the month and day. There logs related to API and Clusters that you can find also in /var/ossec/logs.

These files (which you aim to transfer in real time to S3) should be managed by an AWS data transfer system, as this functionality is not native to Wazuh. After my research, I have found that the most suitable solution for real-time transfer would be to use AWS DataSync. Here is the link to the official AWS documentation: https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html

I have also come across this article: https://michaelsambol.medium.com/move-millions-of-files-from-amazon-ec2-to-amazon-s3-using-aws-datasync-a15bb31a81a1

These are the most immediate solutions I found during my research. If you would like us to implement these together or if you want to explore other solutions, I am here and available to assist. If you're willing to share your code and would like us to explore the solution you've been working on together, I would appreciate a more detailed explanation of your approach. I commit to working on finding a solution with you.

Antonio

[maurya poonam]

Could you explain me all process step by step so that I can move directory log to s3 bucket? 

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bdc9be47-02cf-4572-9eff-25bd0b65502cn%40googlegroups.com.

[Antonio Kim (Wazuh)]

Hi Maurya Poonam!

To send real-time information from a directory on an EC2 instance to an Amazon S3 bucket using AWS DataSync, you should follow these general steps:

1. **Set Up AWS DataSync**:
   - Go to the AWS Management Console.
   - Search for "DataSync" and select it.

2. **Create a DataSync Task**:
   - Click on "Create Task".
   - Provide a name for the task.
   - Choose the source location, which in this case will be the folder on your EC2 instance. This can be done either using a DataSync agent or through NFS.

3. **Configure Source Location**:
   - If you are using a DataSync agent on your EC2 instance, you need to install the agent and configure the source location with the path to the folder you want to synchronize.
   - If you are using NFS, you need to set up NFS on your EC2 instance and then provide the access details in the source location configuration. It is possible that you will need to create and configure a VPC.

4. **Choose Destination Location**:
   - Select "Amazon S3" as the destination location.
   - Choose the S3 bucket where you want to send the data.
   - You can configure advanced options according to your needs.

5. **Schedule the Sync Task**:
   - Set up the schedule for the DataSync task according to your preference. You can configure it to be real-time or schedule it to run at specific intervals.

6. **Review and Create the Task**:
   - Review the task configuration to ensure everything is set up correctly.
   - Click on "Create Task" to start the synchronization.

7. **Monitor and Verify Syncing**:
   - Once the DataSync task is running, you can monitor its progress and verify that data is being synchronized in real-time.

It's important to note that, depending on your requirements, you might want to set up additional options such as access control policies (IAM) and event notifications to manage and monitor data synchronization.
Keep in mind that AWS DataSync is a high-speed, efficient data transfer solution, but costs can vary based on the amount of data transferred and the number of synchronization tasks in progress, so you should consider the DataSync pricing structure based on your needs.

Feel free to use the reference link I shared with you here where you can find a more detailed step-by-step guide.

If you have any questions, I'm here to help with whatever you need.

Antonio