gcp pubsub error

[Fawwas Hamdi]

Hello guys i hope someone can help me with this error 

Error: Could not update configuration (1908) - Error validating configuration: (1230): Invalid element in the configuration: 'project_id'., (1202): Configuration error at 'etc/ossec.conf'.

this is the code that im trying to put within the ossec.conf file 

<gcp-pubsub>
    <project_id>wazuh-dev</project_id>
    <subscription_name>wazuhdns</subscription_name>
    <credentials_file>/etc/credentials.json</credentials_file>
   </gcp-pubsub>

[Fawwas Hamdi]

here some results regarding the gcp-pubsub

[190722 user]# grep gcp-pubsub /var/ossec/logs/ossec.log
2022/07/26 06:59:31 wazuh-modulesd: ERROR: Unknown module 'gcp-pubsub'
2022/07/26 06:59:32 wazuh-modulesd: ERROR: Unknown module 'gcp-pubsub'
2022/07/26 06:59:40 wazuh-modulesd: ERROR: Unknown module 'gcp-pubsub'
2022/07/26 06:59:43 wazuh-modulesd: ERROR: Unknown module 'gcp-pubsub'
2022/07/26 06:59:59 wazuh-modulesd: ERROR: Unknown module 'gcp-pubsub'

[carlos...@wazuh.com]

Hi,

The configuration you shared looks fine. I have tested it both on Wazuh 4.2 and 4.3 and it works. Please confirm that the configuration you have shared with us is exactly the same as the one you have in your ossec.conf. Also, make sure you have entered this configuration inside the "ossec_config" tag in the ossec.conf. Here is an example:

You can place anywhere as long as it's inside the ossec_config tag.

If you are sure of both things and you still get this error please confirm which version of Wazuh you are using and share with us the complete ossec.conf file. You can attach it as a file, but please remove any sensitive information you have entered in it.

I hope it will be helpful.

[Fawwas Hamdi]

as you can see here i can't even save the config I'm using the newest version "WAZUH_VERSION":"v4.3.6"

[Fawwas Hamdi]

im using the ova file for this installation 

[carlos...@wazuh.com]

The only thing I can see that can justify this error is that the credentials file does not exist.. You are using a absolute path to indicate where to find the credentials. Please make sure there is a file called "credentials.json" in "/etc". If you want to use a relative path so you can store your credentials in "Wazuh-path/etc/credentials.json"  then you need to specify it as etc/credentials.json.

[Fawwas Hamdi]

thank you it already did what you're suggested but there is still error

2022/07/26 08:11:40 wazuh-modulesd:gcp-pubsub: INFO: Module started.
2022/07/26 08:11:48 wazuh-modulesd:gcp-pubsub: WARNING: Command returned exit code 1
2022/07/26 08:11:48 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.
2022/07/26 08:12:41 wazuh-modulesd:gcp-pubsub: WARNING: Command returned exit code 1
2022/07/26 08:12:41 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.
2022/07/26 08:13:42 wazuh-modulesd:gcp-pubsub: WARNING: Command returned exit code 1
2022/07/26 08:13:42 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.

[carlos...@wazuh.com]

Thank you for your patience.

Ok, what I see are 2 different errors. On the one hand, the issue that the configuration validator was giving you an error and on the other hand, that the module itself seems to be failing. Can you confirm me if the first problem, the error in the configuration validator, has been solved?

Regarding the other topic, I will help you to debug this, but first confirm me if you have access to the host and the possibility to execute commands on it.

With this information I will be able to help you better.

[Fawwas Hamdi]

yes the first error is solved 

only this 

2022/07/26 08:55:42 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.
2022/07/26 08:56:41 wazuh-modulesd:gcp-pubsub: WARNING: Command returned exit code 1
2022/07/26 08:56:41 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.
2022/07/26 08:57:42 wazuh-modulesd:gcp-pubsub: WARNING: Command returned exit code 1
2022/07/26 08:57:42 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.
2022/07/26 08:58:25 wazuh-modulesd:gcp-pubsub: INFO: Module started.
2022/07/26 08:58:30 wazuh-modulesd:gcp-pubsub: WARNING: Command returned exit code 1
2022/07/26 08:58:30 wazuh-modulesd:gcp-pubsub: WARNING: Unknown error.

[carlos...@wazuh.com]

It is possible to obtain additional information about the module's execution by enabling the debug mode. To enable this you need to access the host and add the following line to the WAZUH_PATH/etc/local_internal_options.conf file and then restart the Wazuh service to enable debug mode:


wazuh_modules.debug=2


Once you enable debug mode and restart the service you will be able to see these logs both from the host or from the UI.

Don't forget to disable debug mode once the troubleshooting has finished. Leaving debug mode enabled could result in the addition of large amounts of logs in the ossec.log file.

[Fawwas Hamdi]

Here is the results

2022/07/26 09:16:41 wazuh-modulesd[10762] main.c:94 at main(): DEBUG: Created new thread for the 'gcp-pubsub' module.
2022/07/26 09:16:41 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:103 at wm_gcp_pubsub_main(): INFO: Module started.
2022/07/26 09:16:41 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:119 at wm_gcp_pubsub_main(): DEBUG: Starting fetching of logs.
2022/07/26 09:16:41 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:207 at wm_gcp_pubsub_run(): DEBUG: Create argument list
2022/07/26 09:16:42 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:259 at wm_gcp_pubsub_run(): DEBUG: Launching command: wodles/gcloud/gcloud --integration_type pubsub --project  --subscription_id wazuh-development-subs --credentials_file /etc/25072021.json --max_messages 100 --num_threads 1 --log_level 2
2022/07/26 09:16:44 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:272 at wm_gcp_pubsub_run(): WARNING: Command returned exit code 1
2022/07/26 09:16:44 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:276 at wm_gcp_pubsub_run(): WARNING: Unknown error.
2022/07/26 09:16:44 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:296 at wm_gcp_pubsub_run(): DEBUG: OUTPUT: gcloud_wodle - CRITICAL - An exception happened while running the wodle: ERROR: No permissions for executing the wodle from this subscription
2022/07/26 09:16:44 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:123 at wm_gcp_pubsub_main(): DEBUG: Fetching logs finished.
2022/07/26 09:16:44 wazuh-modulesd:gcp-pubsub[10762] wm_gcp.c:115 at wm_gcp_pubsub_main(): DEBUG: Sleeping until: 2022/07/26 10:16:41

[carlos...@wazuh.com]

As you can see the issue is that those credentials file does not provide enought permissions to operate with your project. You need to provide Pub/Sub Publisher and Pub/Sub Subscriber permissions.

Please take a look to the following links to learn more about how to configure pubsub and how to set up the credentials:

• Configuring PubSub
• Setting up credentials

Let us know if you need further assistance.

[Fawwas Hamdi]

thank you for your help ill update you later 

[Fawwas Hamdi]

2022/07/27 08:12:20 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:119 at wm_gcp_pubsub_main(): DEBUG: Starting fetching of logs.
2022/07/27 08:12:20 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:207 at wm_gcp_pubsub_run(): DEBUG: Create argument list
2022/07/27 08:12:20 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:259 at wm_gcp_pubsub_run(): DEBUG: Launching command: wodles/gcloud/gcloud --integration_type pubsub --25072021 --subscription_id wazuh-development-subs --credentials_file /etc/25072021-d4148e3ac39a.json --max_messages 100 --num_threads 1 --log_level 1
2022/07/27 08:12:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:272 at wm_gcp_pubsub_run(): WARNING: Command returned exit code 1
2022/07/27 08:12:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:276 at wm_gcp_pubsub_run(): WARNING: Unknown error.
2022/07/27 08:12:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:296 at wm_gcp_pubsub_run(): DEBUG: OUTPUT: gcloud_wodle - DEBUG - Setting 1 thread to pull 100 messages in total
2022/07/27 08:12:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:309 at wm_gcp_pubsub_run(): DEBUG: gcloud_wodle - DEBUG - Setting 1 thread to pull 100 messages in total
2022/07/27 08:12:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:123 at wm_gcp_pubsub_main(): DEBUG: Fetching logs finished.
2022/07/27 08:12:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:115 at wm_gcp_pubsub_main(): DEBUG: Sleeping until: 2022/07/27 08:13:20

I think it is working since the permission message disappeared, but I haven't seen any GCP log within the GCP modules maybe because of the warning message?

2022/07/27 07:41:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:272 at wm_gcp_pubsub_run(): WARNING: Command returned exit code 1
2022/07/27 07:41:22 wazuh-modulesd:gcp-pubsub[15317] wm_gcp.c:276 at wm_gcp_pubsub_run(): WARNING: Unknown error

[carlos...@wazuh.com]

Sorry for the late response.

Your issue seems to be that the module is still failing. Probably due to a permission or configuration issue. However, as you have not set the loggin level to debug there is no information to determine the cause of the issue. I suggest you to enable debug mode and set the GCP logging configuration to debug.


If you have not yet enabled debug mode you can do so by adding the following line to the WAZUH_PATH/etc/local_internal_options.conf file and then restart the Wazuh service to enable debug mode:

wazuh_modules.debug=2


To set logging level to debug you need to add <logging>debug</logging> to your configuration. Here is an example:

<gcp-pubsub>
    <logging>debug</logging>

    <project_id>wazuh-dev</project_id>
    <subscription_name>wazuhdns</subscription_name>
    <credentials_file>/etc/credentials.json</credentials_file>
</gcp-pubsub>

Once you enable debug mode and set the logging level to debug restart the service and check the output againt. It should now contain additional information about what's happening.

Don't forget to disable debug mode once the troubleshooting has finished. Leaving debug mode enabled could result in the addition of large amounts of logs in the ossec.log file.

[Fawwas Hamdi]

<gcp-pubsub>
    <pull_on_start>yes</pull_on_start>
    <interval>1m</interval>
    <logging>debug</logging>
    <project_id>25072021</project_id>
    <subscription_name>wazuh-development-subs</subscription_name>
    <credentials_file>/etc/25072021-d4148e3ac39a.json</credentials_file>
 </gcp-pubsub>

here is my gcp configuraiton and i already set wazuh_modules.debug=2  

2022/08/01 00:36:22 wazuh-modulesd[16349] main.c:94 at main(): DEBUG: Created new thread for the 'gcp-pubsub' module.
2022/08/01 00:36:22 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:103 at wm_gcp_pubsub_main(): INFO: Module started.
2022/08/01 00:36:22 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:119 at wm_gcp_pubsub_main(): DEBUG: Starting fetching of logs.
2022/08/01 00:36:22 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:207 at wm_gcp_pubsub_run(): DEBUG: Create argument list
2022/08/01 00:36:22 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:259 at wm_gcp_pubsub_run(): DEBUG: Launching command: wodles/gcloud/gcloud --integration_type pubsub --25072021 --subscription_id wazuh-development-subs --credentials_file /etc/25072021-d4148e3ac39a.json --max_messages 100 --num_threads 1 --log_level 1
2022/08/01 00:36:30 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:272 at wm_gcp_pubsub_run(): WARNING: Command returned exit code 1
2022/08/01 00:36:30 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:276 at wm_gcp_pubsub_run(): WARNING: Unknown error.
2022/08/01 00:36:30 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:296 at wm_gcp_pubsub_run(): DEBUG: OUTPUT: gcloud_wodle - DEBUG - Setting 1 thread to pull 100 messages in total
2022/08/01 00:36:30 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:309 at wm_gcp_pubsub_run(): DEBUG: gcloud_wodle - DEBUG - Setting 1 thread to pull 100 messages in total
2022/08/01 00:36:30 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:123 at wm_gcp_pubsub_main(): DEBUG: Fetching logs finished.
2022/08/01 00:36:30 wazuh-modulesd:gcp-pubsub[16349] wm_gcp.c:115 at wm_gcp_pubsub_main(): DEBUG: Sleeping until: 2022/08/01 00:37:22

[carlos...@wazuh.com]

I have been looking into this and everything keeps pointing to it being a credential problem. I suggest the following:

• Verify that the project ID you entered in the configuration is correct. You can find this information under Project Info in your dashboard. Make sure you enter the full name. Take a look to the attached screenshot.

• Verify that the credentials json file you are using is correct. Open this file and make sure that the "project_id" that appears has the same value as the one you have set in your config. The same as the one in the Project Info. 

• Make sure that the Service Account you are authenticating with has the roles Pub/Sub Publisher and Pub/Sub Subscriber assigned. Take a look at this page to learn more about how to configure the credentials.

• Finally, make sure that the subscription name is correct, complete and your Service Account has access to it.

[Fawwas Hamdi]

can you provide me the link to access this information? 

• Make sure that the Service Account you are authenticating with has the roles Pub/Sub Publisher and Pub/Sub Subscriber assigned. Take a look at this page to learn more about how to configure the credentials. 

as for right now, I still have the same problem as the last time, even i already make new pubsub configuration with new key still the same issue.