SLACK integration stopped working

[simcha...@dplwireless.com]

SLACK integrations stopped working, I've enabled debugging on integratord and get the following logs when an alert is generated.

2018/10/31 10:54:22 ossec-integratord: DEBUG: file /tmp/slack-1540994062--270073136.alert was written.

2018/10/31 10:54:22 ossec-integratord: DEBUG: Running: /var/ossec/integrations/slack '/tmp/slack-1540994062--270073136.alert' '' 'https://hooks.slack.com/services/REDACTED' > /dev/null 2>&1

2018/10/31 10:54:23 ossec-integratord: ERROR: Unable to run integration for slack -> /var/ossec/integrations/slack

Please advise.

Thanks!

[Miguel Ruiz]

Hi simcha.attis,

I would like to have more information about the error. The Slack integration was running (and you were receiving alerts in Slack) when it stopped or do you see this error right after restarting?

It would be helpful if you can give us more information about your system: Wazuh version(you can check your version in the file /etc/ossec-init.conf), OS, the <integration> section of your configuration at /var/ossec/etc/ossec.conf.

With that information we might be able to give you more accurate help to solve the problem

We have an issue in our repository to improve the debug output for the integration module to make easier to troubleshoot.

Here you have a link to the issue: https://github.com/wazuh/wazuh/issues/1298

Best regards,

Miguel

[simcha...@dplwireless.com]

This works fine and sends to slack when I export a JSON event to /tmp/slack.alert and manually run it --

/var/ossec/integrations/slack '/tmp/slack.alert' '' 'https://hooks.slack.com/services/REDACTED

In the logs it shows the following -

2018/10/31 10:52:33 ossec-integratord: DEBUG: file /tmp/slack-1540993953--546543517.alert was written.

2018/10/31 10:52:33 ossec-integratord: DEBUG: Running: /var/ossec/integrations/slack '/tmp/slack-1540993953--546543517.alert' '' 'https://hooks.slack.com/services/REDACTED' > /dev/null 2>&1

2018/10/31 10:52:35 ossec-integratord: ERROR: Unable to run integration for slack -> /var/ossec/integrations/slack

DIRECTORY="/var/ossec"

NAME="Wazuh"

VERSION="v3.6.1"

REVISION="3608"

DATE="Tue Sep 18 11:31:06 ADT 2018"

TYPE="server"

I'm running on Ubuntu 16.04-5.  Yes this was working prior. 

This is my configuration

  <integration>

     <name>slack</name>

     <hook_url>https://hooks.slack.com/services/REDACTED</hook_url>

     <group>slackapi,</group>

     <max_log>1024</max_log>

     <alert_format>json</alert_format>

  </integration>

Thanks in advance,

[Miguel Ruiz]

Hi simch.attis,

Your configuration looks correct, and it's weird that the integration stops working for a specific alert.

Can you modify the Slack integration script at /var/ossec/integrations/slack?

In #Global vars (line 31) set debug_enabled = True.

This will generate a log file at /var/ossec/logs/integration.log, where we can extract more information to solve the problem.

Restart the manager and check the log file if the error happens again.

Let me know if this helped.

Regards,

Miguel

[simcha...@dplwireless.com]

It didn't write anything to that file however wrote the following in ossec.log

2018/10/31 12:02:00 ossec-integratord: ERROR: Unable to run integration for slack -> /var/ossec/integrations/slack

So frustrating.

[Miguel Ruiz]

Hi simcha.attis,

That is the debug output generated by enabling integrator.debug=1 in the internal_options configuration.

We are working on improving the information generated by enabling that option for future releases.

Setting debug_enabled=True in the /var/ossec/integrations/slack script will generate a new log file with information about the execution of the script, with much more detail than the integrator.debug option.

That way we can check what is causing the error and solve it.

Sorry for the inconvenience.

[simcha...@dplwireless.com]

debug_enabled = True

pwd = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))

json_alert = {}

now = time.strftime("%a %b %d %H:%M:%S %Z %Y")

I do have it set to debug, I also have restarted the manager, and I'm still not getting anything logged to that file.

[Miguel Ruiz]

Hi simcha.attis,

Sorry if I missunderstood.

Did you mean that you saw this log block again at ossec.log and the integration.log is not created?

2018/10/31 10:54:22 ossec-integratord: DEBUG: file /tmp/slack-1540994062--270073136.alert was written.

2018/10/31 10:54:22 ossec-integratord: DEBUG: Running: /var/ossec/integrations/slack '/tmp/slack-1540994062--270073136.alert' '' 'https://hooks.slack.com/services/REDACTED' > /dev/null 2>&1

2018/10/31 10:54:23 ossec-integratord: ERROR: Unable to run integration for slack -> /var/ossec/integrations/slack

If that is the case, please let me know

On Wednesday, October 31, 2018 at 2:58:48 PM UTC+1, simcha...@dplwireless.com wrote:

[simcha...@dplwireless.com]

Yes, that's correct, it didn't create the log file for integration.log. 

[Miguel Ruiz]

The log file is created during the execution of the script.

If the log file is not being created, probably the integratordis not able to even execute it.

Can you verify the permissions for the script are correct executing

chown root:ossec /var/ossec/integrations/slack
chmod 750 /var/ossec/integrations/slack

Then restart the manager again and check if the problem persists

systemctl restart wazuh-manager

Also, make sure you only have one integratord process running executing

ps aux | grep integratord

Let me know if this worked.

Best regards,

Miguel

On Wednesday, October 31, 2018 at 2:58:48 PM UTC+1, simcha...@dplwireless.com wrote:

[simcha...@dplwireless.com]

root@servername:/var/ossec/integrations# ls -la

total 28

drwxr-x---  2 root ossec 4096 Oct 31 13:59 .

drwxr-x--- 22 root ossec 4096 Oct 29 11:24 ..

-rwxr-x---  1 root ossec 1343 Sep  7 12:06 pagerduty

-rwxr-x---  1 root ossec 3389 Oct 31 13:59 slack

-rwxr-x---  1 root ossec 6354 Sep  7 12:06 virustotal

root@servername:/var/ossec/integrations# chmod 750 slack

root@servername:/var/ossec/integrations# ls -la

total 28

drwxr-x---  2 root ossec 4096 Oct 31 13:59 .

drwxr-x--- 22 root ossec 4096 Oct 29 11:24 ..

-rwxr-x---  1 root ossec 1343 Sep  7 12:06 pagerduty

-rwxr-x---  1 root ossec 3389 Oct 31 13:59 slack

-rwxr-x---  1 root ossec 6354 Sep  7 12:06 virustotal

root@servername:/var/ossec/integrations# chown root:ossec /var/ossec/integrations/slack

root@servername:/var/ossec/integrations# systemctl restart wazuh-manager

root@servername:/var/ossec/integrations# ps aux | grep integratord

ossecm    3165  0.0  0.0  28092  2772 ?        S    15:06   0:00 /var/ossec/bin/ossec-integratord

root      3527  0.0  0.0  14224   972 pts/0    S+   15:06   0:00 grep --color=auto integratord

root@servername:/var/ossec/integrations# groups ossecm

ossecm : ossec

Everything looks correct to me.

[simcha...@dplwireless.com]

Looks like python-requests and python3-requests modules updated themselves on Oct19, but it seems this was working on the Oct 22, would it somehow be incompatible with the latest versions?

python3-requests is already the newest version (2.9.1-3ubuntu0.1).

python-requests is already the newest version (2.9.1-3ubuntu0.1).

Let me know I'd like to get it resolved ASAP.

Thanks in advance.

[simcha...@dplwireless.com]

Actually, can't be the requests module upgrade as it actually works fine when I run it manually at the command prompt. 

[simcha...@dplwireless.com]

Have you since been able to replicate this bug?  This is a huge feature of the system that we use to use heavily which is now failing us.. 

[juancarl...@wazuh.com]

Hello,

I tried replicating as closely as possible the behavior you observed. I am also using for this Ubuntu 16.04.5 and python-requests 2.9.1-3ubuntu0.1

One way of getting a similar error is by adding a space between the <hook_url> tag and the https of the URL. I see from what you paste that this is not you exact case but it shows that it could be as simple as a broken url. In case you want to do a quick test you can run the command:          

grep -P 'https://hooks.slack.*(?=<)' /var/ossec/etc/ossec.conf -o | xargs -I % /var/ossec/integrations/slack '/tmp/slack.alert' '' '%'

Since the script worked manually before this should work if that part of the configuration is correct.

Due to the lack of verbosity in the debug I would suggest to make sure to be working with a clean version of the slack integration script, to avoid running blindly into other issues. So if you run:

wget https://raw.githubusercontent.com/wazuh/wazuh/3.6/integrations/slack -O /var/ossec/integrations/slack && chmod 750 /var/ossec/integrations/slack && chown root:ossec /var/ossec/integrations/slack

You should be able to get back to the original file.

I also see that by setting integrator.debug 2 in the internal_options.conf instead of just 1 ( as per the documentation: https://documentation.wazuh.com/current/user-manual/reference/internal-options.html#integrator ) I do observe the /var/ossec/integrations.log file, however the log is not much verbose until you change debug_enabled=True in the /var/ossec/integrations/slack file.

Can you try that and let me know if you get a more verbose log in /var/ossec/logs/integrations.log?

Kind regards,
Juan Carlos

[simcha...@dplwireless.com]

I made these changes and I'm still not able to get logs or a slack alert.. 

[simcha...@dplwireless.com]

I have resolved these issues.. it seems upon further investigation that the log file for integrations.log had the wrong permissions and was causing the module to fail.

I resolved the permissions and all is working as expected.

Thanks everyone. 

[juancarl...@wazuh.com]

Hello,

Thank you for letting us know and contributing to the mailing list, so people who may face similar issues in the future may find help here.

We'll be very glad to continue helping you in the future,

Best regards,

Juan Carlos

On Monday, November 5, 2018 at 11:23:52 PM UTC+1,  wrote:

I have resolved these issues.. it seems upon further investigation that the log file for integrations.log had the wrong permissions and was causing the module to fail.

I resolved the permissions and all is working as expected.

Thanks everyone. 

[simcha...@dplwireless.com]

Maybe you can add some validation for log file permissions to the validate script to catch this before it causes a problem for some people!! :)

[juancarl...@wazuh.com]

Hello,
I do wonder how common it might be for the file permissions of the log file to be changed. Do you know why it happened in your case?
​
An improvement for a future version is already in place with this pull request: https://github.com/wazuh/wazuh/pull/1769 which aims to fix the issue that Miguel previously mentioned regarding the lack of information to debug the integration.
​
Hopefully that will make it a lot easier to diagnose in the future.
​
Best regards,
Juan Carlos Tello

On Tuesday, November 6, 2018 at 9:20:23 PM UTC+1, :

[simcha...@dplwireless.com]

I actually investigated further and logrotate configuration file was setting file permissions to 640 instead of 660.. I modified this and the problem doesn't reoccur.. Can you ensure your configuration file in the release doesn't have 640 as the permissions as this will overwrite on upgrade and may affect other users.

Thanks!

[alberto....@wazuh.com]

Hello

  Sorry for the late response. Yes, we finally reviewed the permissions. Thanks a lot for helping us!

Best regards,