URLHaus Integration Error

[Nico Alonso]

Hi, I followed all the indications of this blog https://wazuh.com/blog/detecting-malicious-urls-using-wazuh-and-urlhaus/ and when I test the integration I always receive this error:

2023/11/07 15:06:03 wazuh-integratord: ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699369563--1791198033.alert  https://urlhaus-api.abuse.ch/v1/url/  > /dev/null 2>&1). Check file and permissions.
2023/11/07 15:06:03 wazuh-integratord: ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699369563-547701340.alert  https://urlhaus-api.abuse.ch/v1/url/  > /dev/null 2>&1). Check file and permissions.
2023/11/07 15:06:13 wazuh-integratord: ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699369573--1876344460.alert  https://urlhaus-api.abuse.ch/v1/url/  > /dev/null 2>&1). Check file and permissions.
2023/11/07 15:06:13 wazuh-integratord: ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699369573--212907064.alert  https://urlhaus-api.abuse.ch/v1/url/  > /dev/null 2>&1). Check file and permissions.
2023/11/07 15:06:15 wazuh-integratord: ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699369575--1512340807.alert  https://urlhaus-api.abuse.ch/v1/url/  > /dev/null 2>&1). Check file and permissions.

I saw another thread where another person had the same problem as me, and I took the custom-urlhaus.py file that was given to them, but it doesnt fix my problem.

I upload my custom-urlhaus.py, all the configuration is the same as the one in the blog, all kind of help is welcome, thanks!

[Christian Borla]

Hello Nico,
I hope you are doing well!

Please double check the permissions

chmod 750 /var/ossec/integrations/custom-urlhaus.py
chown root:ossec /var/ossec/integrations/custom-urlhaus.py

Have you tried running it manually?
Only the python script to run it manually. Running it with some fake values, also you can create an a valid alert json in /var/ossec/integrations/alert.json.

/var/ossec/framework/python/bin/python3 /var/ossec/integrations/custom-urlhaus.py /var/ossec/integrations/alert.json testkey testurl

The idea is check if everithing regardin the script it's ok, also you can use your real key and url, event should be sent to  urlhaus.
Let me know if that works.
Regards.

[suricata]

Hí, Christian.

root:ossec or root:wazuh ?

Regards,

[Nico Alonso]

Hi, thanks for replying!

The permissions were the indicated, but with root:wazuh instead of root:ossec, as it said in the documentation.

I tried to run the script manually and it runs OK. I ran it with de debug flag activated, the command I used is /var/ossec/framework/python/bin/python3 /var/ossec/integrations/custom-urlhaus.py /var/ossec/integrations/alert.json a a debug  and the content of the file alert.json is the following, simply using the comand curl http://pastebin.com/raw/ZkwP7zPF in an agent, where the URL is a positive case for URLHaus:

{"timestamp":"2023-11-08T07:18:38.227+0000","rule":{"level":3,"description":"Suricata: Alert - ET POLICY curl User-Agent Outbound","id":"86601","firedtimes":167,"mail":false,"groups":["ids","suricata"]},"agent":{"id":"005","name":"suricata","ip":"172.30.103.42"},"manager":{"name":"wazuh-server"},"id":"1699427918.9919257","decoder":{"name":"json"},"data":{"timestamp":"2023-11-08T07:18:38.127337+0000","flow_id":"1893192943492407.000000","in_iface":"ens5","event_type":"alert","src_ip":"172.30.103.42","src_port":"57460","dest_ip":"104.20.67.143","dest_port":"80","proto":"TCP","pkt_src":"wire/pcap","tx_id":"0","alert":{"action":"allowed","gid":"1","signature_id":"2013028","rev":"7","signature":"ET POLICY curl User-Agent Outbound","category":"Attempted Information Leak","severity":"2","metadata":{"created_at":["2011_06_14"],"updated_at":["2022_05_03"]}},"http":{"hostname":"pastebin.com","url":"/raw/ZkwP7zPF","http_user_agent":"curl/8.3.0","http_method":"GET","protocol":"HTTP/1.1","status":"301","redirect":"https://pastebin.com/raw/ZkwP7zPF","length":"3"},"app_proto":"http","direction":"to_server","flow":{"pkts_toserver":"4","pkts_toclient":"3","bytes_toserver":"359","bytes_toclient":"498","start":"2023-11-08T07:18:38.113113+0000","src_ip":"172.30.103.42","dest_ip":"104.20.67.143","src_port":"57460","dest_port":"80"}},"location":"/var/log/suricata/eve.json"}

I paste the result of running the script manually with de debug flag activated:

Wed Nov 08 07:54:16 UTC 2023 /var/ossec/integrations/alert.json a a debug
Wed Nov 08 07:54:16 UTC 2023: # Starting
Wed Nov 08 07:54:16 UTC 2023: # File location
Wed Nov 08 07:54:16 UTC 2023: /var/ossec/integrations/alert.json
Wed Nov 08 07:54:16 UTC 2023: # Processing alert
Wed Nov 08 07:54:16 UTC 2023: {'timestamp': '2023-11-08T07:18:38.227+0000', 'rule': {'level': 3, 'description': 'Suricata: Alert - ET POLICY curl User-Agent Outbound', 'id': '86601', 'firedtimes': 167, 'mail': False, 'groups': ['ids', 'suricata']}, 'agent': {'id': '005', 'name': 'suricata', 'ip': '172.30.103.42'}, 'manager': {'name': 'wazuh-server'}, 'id': '1699427918.9919257', 'decoder': {'name': 'json'}, 'data': {'timestamp': '2023-11-08T07:18:38.127337+0000', 'flow_id': '1893192943492407.000000', 'in_iface': 'ens5', 'event_type': 'alert', 'src_ip': '172.30.103.42', 'src_port': '57460', 'dest_ip': '104.20.67.143', 'dest_port': '80', 'proto': 'TCP', 'pkt_src': 'wire/pcap', 'tx_id': '0', 'alert': {'action': 'allowed', 'gid': '1', 'signature_id': '2013028', 'rev': '7', 'signature': 'ET POLICY curl User-Agent Outbound', 'category': 'Attempted Information Leak', 'severity': '2', 'metadata': {'created_at': ['2011_06_14'], 'updated_at': ['2022_05_03']}}, 'http': {'hostname': 'pastebin.com', 'url': '/raw/ZkwP7zPF', 'http_user_agent': 'curl/8.3.0', 'http_method': 'GET', 'protocol': 'HTTP/1.1', 'status': '301', 'redirect': 'https://pastebin.com/raw/ZkwP7zPF', 'length': '3'}, 'app_proto': 'http', 'direction': 'to_server', 'flow': {'pkts_toserver': '4', 'pkts_toclient': '3', 'bytes_toserver': '359', 'bytes_toclient': '498', 'start': '2023-11-08T07:18:38.113113+0000', 'src_ip': '172.30.103.42', 'dest_ip': '104.20.67.143', 'src_port': '57460', 'dest_port': '80'}}, 'location': '/var/log/suricata/eve.json'}
Wed Nov 08 07:54:16 UTC 2023: {'query_status': 'ok', 'id': '2045738', 'urlhaus_reference': 'https://urlhaus.abuse.ch/url/2045738/', 'url': 'https://pastebin.com/raw/ZkwP7zPF', 'url_status': 'offline', 'host': 'pastebin.com', 'date_added': '2022-02-16 21:28:04 UTC', 'last_online': '2022-04-13 22:XX:XX UTC', 'threat': 'malware_download', 'blacklists': {'spamhaus_dbl': 'not listed', 'surbl': 'not listed'}, 'reporter': 'pmelson', 'larted': 'true', 'takedown_time_seconds': '50110124', 'tags': ['PowerShellSMTPInfoStealer'], 'payloads': [{'firstseen': '2022-02-16', 'filename': None, 'file_type': 'txt', 'response_size': '1186', 'response_md5': '9837238a94e0aacb1186fa7cfe97f671', 'response_sha256': 'ae6ac2c0135531cf68c7546e663b8f02b4e43be6a8a0b0faf256ec9d385d2545', 'urlhaus_download': 'https://urlhaus-api.abuse.ch/v1/download/ae6ac2c0135531cf68c7546e663b8f02b4e43be6a8a0b0faf256ec9d385d2545/', 'signature': None, 'virustotal': None, 'imphash': None, 'ssdeep': '24:+U1HHnkr/EuokcV5SgTs3Dvoxu+rEJ79KZCgrcYCe:+Knt/5SgIz4DEJE3rcYCe', 'tlsh': 'T1EC21F124D398A0604669B797F262BC02690C059E1DF1F6644BDBE8AF41CFB846224E'}]}
Wed Nov 08 07:54:16 UTC 2023: ok
Wed Nov 08 07:54:16 UTC 2023: {'urlhaus': {'found': 1, 'source': {'alert_id': '1699427918.9919257', 'rule': '86601', 'description': 'Suricata: Alert - ET POLICY curl User-Agent Outbound', 'url': 'https://pastebin.com/raw/ZkwP7zPF'}, 'urlhaus_reference': 'https://urlhaus.abuse.ch/url/2045738/', 'url_status': 'offline', 'url_date_added': '2022-02-16 21:28:04 UTC', 'url_threat': 'malware_download', 'url_blacklist_spamhaus': 'not listed', 'url_blacklist_surbl': 'not listed', 'url_tags': ['PowerShellSMTPInfoStealer']}, 'integration': 'custom-urlhaus'}
Wed Nov 08 07:54:16 UTC 2023: 1:[005] (suricata) 172.30.103.42->urlhaus:{"urlhaus": {"found": 1, "source": {"alert_id": "1699427918.9919257", "rule": "86601", "description": "Suricata: Alert - ET POLICY curl User-Agent Outbound", "url": "https://pastebin.com/raw/ZkwP7zPF"}, "urlhaus_reference": "https://urlhaus.abuse.ch/url/2045738/", "url_status": "offline", "url_date_added": "2022-02-16 21:28:04 UTC", "url_threat": "malware_download", "url_blacklist_spamhaus": "not listed", "url_blacklist_surbl": "not listed", "url_tags": ["PowerShellSMTPInfoStealer"]}, "integration": "custom-urlhaus"}
Wed Nov 08 07:54:16 UTC 2023 /tmp/custom-gchat-1699430056-191113913.alert    > /dev/null 2>&1

This script dont need the api-key or hook-url to be sent because it is already in the script, thats why I put the values "a", with another values it works. I also know that is working because I have an integration with Google Meets and a message is sent when an alert with level > 10 is triggered, and the message is sent. 

But when I use the integration it keeps returning the same error, I tried to put values for hook_url and api_key in the ossec.conf but same result.

[Christian Borla]

Hi Nicolas!
As you and meerkat have said, it's root:wazuh, sorry about that.

On the other hand, it's good that it works by running it manually, plus the Python script is already pointing to #!/var/ossec/framework/python/bin/python3.
What we can do is to increase the log level to debug 2 for the integration module in the /var/ossec/etc/internal_options.conf file.

# Integrator daemon debug (server, local or Unix agent)
integrator.debug=2

Then restart the manager and try to run the integration to see what messages we find in /var/ossec/log/ossec.log.

Also I found this link, it is a similar situation. 
Let me know the results.

[Nico Alonso]

Hi again Christian!

I followed your advice and increased the log level but the result in the log is almost the same:

 DEBUG: file /tmp/custom-urlhaus.py-1699456476--1820006352.alert was written.
2023/11/08 15:14:36 wazuh-integratord[25408] integrator.c:403 at OS_IntegratorD(): DEBUG: Running: integrations /tmp/custom-urlhaus.py-1699456476--1820006352.alert   debug
2023/11/08 15:14:36 wazuh-integratord[25408] integrator.c:419 at OS_IntegratorD(): ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699456476--1820006352.alert   debug). Check file and permissions.

This is the block of XML for the integration in /var/osec/ossec.conf:

<integration>
    <name>custom-urlhaus.py</name>
    <rule_id>86601</rule_id>
    <alert_format>json</alert_format>
  </integration>

And I also show the content of my directory /var/ossec/integrations, where the permissions seems to be OK (the alert.json is the file that I used to test manually the script)

Unfortunatelly, the link that you sent dont fix the problem, but thanks for sending ir!

[Christian Borla]

Hi Nico!
Yes, it's strange, we could see if all the files have the correct permissions in the following path /var/ossec/framework/python/lib/python3.9/site-packages/, in this case root:wazuh, it is a test that if it does not work we should revert the case. just in case.

Let me know if that works.

[Nico Alonso]

I checked the files in path /var/ossec/framework/python/lib/python3.9/site-packages/  and all had root:wazuh, the right permissions, and it remains the same:

2023/11/08 15:40:37 wazuh-integratord[29670] integrator.c:276 at OS_IntegratorD(): DEBUG: file /tmp/custom-urlhaus.py-1699458037--898305302.alert was written.
2023/11/08 15:40:37 wazuh-integratord[29670] integrator.c:403 at OS_IntegratorD(): DEBUG: Running: integrations /tmp/custom-urlhaus.py-1699458037--898305302.alert   debug
2023/11/08 15:40:37 wazuh-integratord[29670] integrator.c:419 at OS_IntegratorD(): ERROR: Couldn't execute command (integrations /tmp/custom-urlhaus.py-1699458037--898305302.alert   debug). Check file and permissions.

[Christian Borla]

Hi Nico.
OK, I will ask to others teams and I will try to reproduce it in my environment.
I will back as soon as possible.
Regards.

[Christian Borla]

Hi Nico.
I hope you are doing fine, 
I make it works following the same guide.

1. I created the python script, the same from the guide.

ls -l  /var/ossec/integrations/custom-urlhaus.py
-rwxr-x--- 1 root wazuh 5480 nov  8 11:46 /var/ossec/integrations/custom-urlhaus.py

2. as you can see I also changed the permissions and owner.

chmod 750 /var/ossec/integrations/custom-urlhaus.py
chown root:wazuh /var/ossec/integrations/custom-urlhaus.py

3. Then I updated the configuration of the ossec.conf file in the manager

<integration>
  <name>custom-urlhaus.py</name>
  <hook_url>https://urlhaus-api.abuse.ch/v1/url/</hook_url>

  <rule_id>86601</rule_id>
  <alert_format>json</alert_format>
</integration>

4. I validated that the python I am using is correct

#!/var/ossec/framework/python/bin/python3

5. I updated the ruleset with the new custom rule.

<group name=”local, suricata,”>
  <rule id="100004" level="10">
    <field name="urlhaus.url_threat">malware_download</field>
    <description>URLhaus: An endpoint connected to a url known for deploying malware.</description>
  </rule>
</group>

6. Then I restarted the manager.

/var/ossec/bin/wazuh-control restart

7.  At this point I did something different, just to test the integration, as I don't have suricata installed, I created an example log similar to the one generated by suricata, which triggers the rule 86601 which triggers the integration. Then in an external agent I started to monitor a file, in which I paste the log that I have created to be collected by people and sent to the manager, in this way I simulate the circuit (without installing suricata) and I see that the 86601 alert is generated.

In agent side I created the file where I pasted some suricata logs to collect. (it's possible to do the same in manager side)

  <localfile>
      <location>C:\Users\test.txt</location>
      <log_format>syslog</log_format>
  </localfile>


8. When I paste the example log into the test.txt file, the event is collected and the 86601 alert is triggered.

{"timestamp":"2023-11-09T16:19:07.245-0300","rule":{"level":10,"description":"URLhaus: An endpoint connected to a url known for deploying malware.","id":"100004","firedtimes":1,"mail":false,"groups":["local"," s
uricata"]},"agent":{"id":"006","name":"DESKTOP","ip":"192.168.55.100"},"manager":{"name":"VBox"},"id":"1699547.252831","full_log":"{\"urlhaus\": {\"found\": 1, \"source\": {\"alert_id\": \"169955754
4.249813\", \"rule\": \"86601\", \"description\": \"Suricata: Alert - ET POLICY curl User-Agent Outbound\", \"url\": \"https://pastebin.com/raw/ZkwP7zPF\"}, \"urlhaus_reference\": \"https://urlhaus.abuse.ch/url/
2045738/\", \"url_status\": \"offline\", \"url_date_added\": \"2022-02-16 21:28:04 UTC\", \"url_threat\": \"malware_download\", \"url_blacklist_spamhaus\": \"not listed\", \"url_blacklist_surbl\": \"not listed\"
, \"url_tags\": [\"PowerShellSMTPInfoStealer\"]}, \"integration\": \"custom-urlhaus\"}","decoder":{"name":"json"},"data":{"urlhaus":{"found":"1","source":{"alert_id":"1699557544.249813","rule":"86601","descripti
on":"Suricata: Alert - ET POLICY curl User-Agent Outbound","url":"https://pastebin.com/raw/ZkwP7zPF"},"urlhaus_reference":"https://urlhaus.abuse.ch/url/2045738/","url_status":"offline","url_date_added":"2022-02-
16 21:28:04 UTC","url_threat":"malware_download","url_blacklist_spamhaus":"not listed","url_blacklist_surbl":"not listed","url_tags":["PowerShellSMTPInfoStealer"]},"integration":"custom-urlhaus"},"location":"url
haus"}

9. The alert 100004 also is triggered, post processing.

{"timestamp":"2023-11-09T16:19:08.756-0300","rule":{"level":10,"description":"URLhaus: An endpoint connected to a url known for deploying malware.","id":"100004","firedtimes":2,"mail":false,"groups":["local"," s
uricata"]},"agent":{"id":"006","name":"DESKTOP","ip":"192.168.55.100"},"manager":{"name":"VBox"},"id":"1699548.254118","full_log":"{\"urlhaus\": {\"found\": 1, \"source\": {\"alert_id\": \"169955754
4.251322\", \"rule\": \"86601\", \"description\": \"Suricata: Alert - ET POLICY curl User-Agent Outbound\", \"url\": \"https://pastebin.com/raw/ZkwP7zPF\"}, \"urlhaus_reference\": \"https://urlhaus.abuse.ch/url/
2045738/\", \"url_status\": \"offline\", \"url_date_added\": \"2022-02-16 21:28:04 UTC\", \"url_threat\": \"malware_download\", \"url_blacklist_spamhaus\": \"not listed\", \"url_blacklist_surbl\": \"not listed\"
, \"url_tags\": [\"PowerShellSMTPInfoStealer\"]}, \"integration\": \"custom-urlhaus\"}","decoder":{"name":"json"},"data":{"urlhaus":{"found":"1","source":{"alert_id":"1699557544.251322","rule":"86601","descripti
on":"Suricata: Alert - ET POLICY curl User-Agent Outbound","url":"https://pastebin.com/raw/ZkwP7zPF"},"urlhaus_reference":"https://urlhaus.abuse.ch/url/2045738/","url_status":"offline","url_date_added":"2022-02-
16 21:28:04 UTC","url_threat":"malware_download","url_blacklist_spamhaus":"not listed","url_blacklist_surbl":"not listed","url_tags":["PowerShellSMTPInfoStealer"]},"integration":"custom-urlhaus"},"location":"url
haus"}

10. The result in /var/ossec/logs/integrations.log 

Thu Nov 09 16:19:05 -03 2023 /tmp/custom-urlhaus.py-1699557545--1960130980.alert  https://urlhaus-api.abuse.ch/v1/url/
Thu Nov 09 16:19:07 -03 2023 /tmp/custom-urlhaus.py-1699557547--1975126914.alert  https://urlhaus-api.abuse.ch/v1/url/

11. The result in /var/ossec/logs/ossec.log (note that the date time is different beacuse I forgot to enbled the debug level in the integration module)

2023/11/09 16:46:01 wazuh-integratord[30732] integrator.c:161 at OS_IntegratorD(): DEBUG: Sending new alert.
2023/11/09 16:46:01 wazuh-integratord[30732] integrator.c:293 at OS_IntegratorD(): DEBUG: File /tmp/custom-urlhaus.py-1699559161--1248223620.alert was written.
2023/11/09 16:46:01 wazuh-integratord[30732] integrator.c:442 at OS_IntegratorD(): DEBUG: Running script with args: integrations /tmp/custom-urlhaus.py-1699559161--1248223620.alert  https://urlhaus-api.abuse.ch/v1/url/ debug
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: # Starting
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: # File location
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: /tmp/custom-urlhaus.py-1699559161--1248223620.alert
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: # Processing alert
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: {'timestamp': .................
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: {'query_status': 'ok', ..........
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: ok
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: {'urlhaus': {'found': 1, .................
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:451 at OS_IntegratorD(): DEBUG: Thu Nov 09 16:46:01 -03 2023: 1:[006] (DESKTOP) 192.168.56.1->urlhaus:{"urlhaus": {"found": 1, "source": ...................
2023/11/09 16:46:02 wazuh-integratord[30732] integrator.c:464 at OS_IntegratorD(): DEBUG: Command ran successfully.

now looking at the logs you got, I wonder if the problem is that wazuh or integratord don't have permissions to create a file in /tmp.
Let me know if taht helps.
Regards.

[Christian Borla]

Erratum: The alert in the point 8 is the following. 

{"timestamp":"2023-11-09T16:19:04.438-0300","rule":{"level":3,"description":"Suricata: Alert - ET POLICY curl User-Agent Outbound","id":"86601","firedtimes":5,"mail":false,"groups":["ids","suricata"]},"agent":{"id":"006","name":"DESKTOP","ip":"192.168.55.100"},"manager":{"name":"VBox"},"id":"1699544.251322","decoder":{"name":"json"},"data":{"timestamp":"2016-05-02T17:46:48.515262+0000","flow_id":"1234","in_iface":"eth0","event_type":"alert","src_ip":"16.10.10.10","src_port":"5555","dest_ip":"16.10.10.11","dest_port":"80","proto":"TCP","alert":{"action":"allowed","gid":"1","signature_id":"2019236","rev":"3","signature":"ET POLICY curl User-Agent Outbound","category":"Attempted Administrator Privilege Gain","severity":"1"},"http":{"hostname":"pastebin.com","url":"/raw/ZkwP7zPF","http_user_agent":"curl/8.3.0","http_method":"GET","protocol":"HTTP/1.1","status":"301","redirect":"https://pastebin.com/raw/ZkwP7zPF","length":"3"},"payload":"abcde","payload_printable":"hi test","stream":"0","host":"suricata.com"},"location":"C:\\Users\\test.txt"}

Regards.

[Nico Alonso]

Hi again Christian, sorry for the late reply!

I was doing again all the steps and this time, after I copied another time the script from the tutorial, now its working...

I dont know what I did different the last time, but now is solved, so thanks for your help! now the integration is working perfect

[Christian Borla]

Great!
You are welcome!