URLHaus integration error

[Hakim Mutalib]

Hi everyone and good day,

I've been trying to integrating my wazuh server with URLHaus and it has been great so far until I stumbled upon this error. I am still a newbie in this field and hope someone can help me to rectify this problem. 

I've attached the debug log below.

[Openime Oniagbi]

Hi,

Thank you for using Wazuh.

What version of Wazuh are you using please?

For version 4.3.0 or later: Confirm you have changed the permissions of the file using:

chmod 750 /var/ossec/integrations/custom-urlhaus.py
chown root:wazuh /var/ossec/integrations/custom-urlhaus.py

For versions before 4.3, you'd have to use:

chmod 750 /var/ossec/integrations/custom-urlhaus.py

chown root:ossec /var/ossec/integrations/custom-urlhaus.py

I hope this helps. Please see this blog for more information.

Regards.

[Hakim Mutalib]

Hi Openime! thanks for replying to my post.

regarding your question about wazuh version, I am using version 4.3.6

and regarding the file permission I have already did the correct way, as you can see below.

[Openime Oniagbi]

Hi,

That looks good.

Please send me your script.

Regards,

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/da835e96-fe5a-4f8c-9914-cc1da7e4f4c6n%40googlegroups.com.

--

Openime Oniagbi THREAT INTELLIGENCE

[Hakim Mutalib]

After running debug mode on, I got this error.

#!/var/ossec/framework/python/bin/python3

# Copyright (C) 2015-2022, Wazuh Inc.

import json

import sys

import time

import os

from socket import socket, AF_UNIX, SOCK_DGRAM

try:

    import requests

    from requests.auth import HTTPBasicAuth

except Exception as e:

    print ("No module 'requests' found. Install: pip install requests")

    sys.exit(1)

# Global vars

debug_enabled = True

pwd = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))

json_alert = {}

now = time.strftime('%a %b %d %H:%M:%S %Z %Y')

# Set paths

log_file = '{0}/logs/integrations.log'.format(pwd)

socket_addr = '{0}/queue/sockets/queue'.format(pwd)

def main(args):

    debug('# Starting')

    # Read args

    alert_file_location = args[1]

    debug('# File location')

    debug(alert_file_location)

    # Load alert. Parse JSON object.

    with open(alert_file_location) as alert_file:

        json_alert = json.load(alert_file)

    debug('# Processing alert')

    debug(json_alert)

    # Request urlhaus info

    msg = request_urlhaus_info(json_alert)

    # If positive match, send event to Wazuh Manager

    if msg:

        send_event(msg, json_alert['agent'])

def debug(msg):

    if debug_enabled:

        msg = '{0}: {1}\n'.format(now, msg)

        print (msg)

        f = open(log_file, 'a')

        f.write(msg)

        f.close()

def collect(data):

    urlhaus_reference = data['urlhaus_reference']

    url_status = data['url_status']

    url_date_added = data['date_added']

    url_threat = data['threat']

    url_blacklist_spamhaus = data['blacklists']['spamhaus_dbl']

    url_blacklist_surbl = data['blacklists']['surbl']

    url_tags = data['tags']

    return (

        urlhaus_reference,

        url_status,

        url_date_added,

        url_threat,

        url_blacklist_spamhaus,

        url_blacklist_surbl,

        url_tags,

        )

def in_database(data, url):

    result = data['query_status']

    debug(result)

    if result == 'ok':

        return True

    return False

def query_api(url):

    params = {'url': url}

    response = requests.post('https://urlhaus-api.abuse.ch/v1/url/',

                             params)

    json_response = response.json()

    if json_response['query_status'] == 'ok':

        data = json_response

        debug(data)

        return data

    else:

        alert_output = {}

        alert_output['urlhaus'] = {}

        alert_output['integration'] = 'custom-urlhaus'

        json_response = response.json()

        debug('# Error: The URLHAUS integration encountered an error')

        alert_output['urlhaus']['error'] = response.status_code

        alert_output['urlhaus']['description'] = json_response['errors'

                ][0]['detail']

        send_event(alert_output)

        exit(0)

def request_urlhaus_info(alert):

    alert_output = {}

    # If there is no url address present in the alert. Exit.

    if alert['data']['http']['redirect'] == None:

        return 0

    # Request info using urlhaus API

    data = query_api(alert['data']['http']['redirect'])

    # Create alert

    alert_output['urlhaus'] = {}

    alert_output['integration'] = 'custom-urlhaus'

    alert_output['urlhaus']['found'] = 0

    alert_output['urlhaus']['source'] = {}

    alert_output['urlhaus']['source']['alert_id'] = alert['id']

    alert_output['urlhaus']['source']['rule'] = alert['rule']['id']

    alert_output['urlhaus']['source']['description'] = alert['rule'

            ]['description']

    alert_output['urlhaus']['source']['url'] = alert['data']['http'

            ]['redirect']

    url = alert['data']['http']['redirect']

    # Check if urlhaus has any info about the url

    if in_database(data, url):

        alert_output['urlhaus']['found'] = 1

    # Info about the url found in urlhaus

    if alert_output['urlhaus']['found'] == 1:

        (

            urlhaus_reference,

            url_status,

            url_date_added,

            url_threat,

            url_blacklist_spamhaus,

            url_blacklist_surbl,

            url_tags,

            ) = collect(data)

        # Populate JSON Output object with urlhaus request

        alert_output['urlhaus']['urlhaus_reference'] = urlhaus_reference

        alert_output['urlhaus']['url_status'] = url_status

        alert_output['urlhaus']['url_date_added'] = url_date_added

        alert_output['urlhaus']['url_threat'] = url_threat

        alert_output['urlhaus']['url_blacklist_spamhaus'] = \

            url_blacklist_spamhaus

        alert_output['urlhaus']['url_blacklist_surbl'] = \

            url_blacklist_surbl

        alert_output['urlhaus']['url_tags'] = url_tags

    debug(alert_output)

    return alert_output

def send_event(msg, agent=None):

    if not agent or agent['id'] == '000':

        string = '1:urlhaus:{0}'.format(json.dumps(msg))

    else:

        string = '1:[{0}] ({1}) {2}->urlhaus:{3}'.format(agent['id'],

                agent['name'], (agent['ip'] if 'ip' in agent else 'any'

                ), json.dumps(msg))

    debug(string)

    sock = socket(AF_UNIX, SOCK_DGRAM)

    sock.connect(socket_addr)

    sock.send(string.encode())

    sock.close()

if __name__ == '__main__':

    try:

        # Read arguments

        bad_arguments = False

        if len(sys.argv) >= 4:

            msg = '{0} {1} {2} {3} {4}'.format(now, sys.argv[1],

                    sys.argv[2], sys.argv[3],

                    (sys.argv[4] if len(sys.argv) > 4 else ''))

            debug_enabled = len(sys.argv) > 4 and sys.argv[4] == 'debug'

        else:

            msg = '{0} Wrong arguments'.format(now)

            bad_arguments = True

        # Logging the call

        f = open(log_file, 'a')

        f.write(msg + '\n')

        f.close()

        if bad_arguments:

            debug('# Exiting: Bad arguments.')

            sys.exit(1)

        # Main function

        main(sys.argv)

    except Exception as e:

        debug(str(e))

        raise

--

Hakim Mutalib

Security Auditor Intern 

RunCloud - Simplifying Server Management

Create . Connect . Deploy

[Openime Oniagbi]

Hi,

I have compared your script and the one on this blog and seen that there are a quite some differences.

I have attached the script from the blog. Can you replace it in the integrations folder, confirm file permissions and restart your Wazuh server and let me know if it works?

Regards,

[Openime Oniagbi]

Please see attached.

[Hakim Mutalib]

Same error occur, 

#!/var/ossec/framework/python/bin/python3

# Copyright (C) 2015-2022, Wazuh Inc.

import json

import sys

import time

import os

from socket import socket, AF_UNIX, SOCK_DGRAM

try:

    import requests

    from requests.auth import HTTPBasicAuth

except Exception as e:

    print("No module 'requests' found. Install: pip install requests")

  return urlhaus_reference, url_status, url_date_added, url_threat, url_blacklist_spamhaus, url_blacklist_surbl, url_tags

      return(0)

    # Request info using urlhaus API

    data = query_api(alert["data"]["http"]["redirect"])

    # Create alert

    alert_output["urlhaus"] = {}

    alert_output["integration"] = "custom-urlhaus"

    alert_output["urlhaus"]["found"] = 0

    alert_output["urlhaus"]["source"] = {}

    alert_output["urlhaus"]["source"]["alert_id"] = alert["id"]

    alert_output["urlhaus"]["source"]["rule"] = alert["rule"]["id"]

    alert_output["urlhaus"]["source"]["description"] = alert["rule"]["description"]

    alert_output["urlhaus"]["source"]["url"] = alert["data"]["http"]["redirect"]

    url = alert["data"]["http"]["redirect"]

    # Check if urlhaus has any info about the url

    if in_database(data, url):

      alert_output["urlhaus"]["found"] = 1

    # Info about the url found in urlhaus

    if alert_output["urlhaus"]["found"] == 1:

        urlhaus_reference, url_status, url_date_added, url_threat, url_blacklist_spamhaus, url_blacklist_surbl, url_tags = collect(data)

        # Populate JSON Output object with urlhaus request

        alert_output["urlhaus"]["urlhaus_reference"] = urlhaus_reference

        alert_output["urlhaus"]["url_status"] = url_status

        alert_output["urlhaus"]["url_date_added"] = url_date_added

        alert_output["urlhaus"]["url_threat"] = url_threat

        alert_output["urlhaus"]["url_blacklist_spamhaus"] = url_blacklist_spamhaus

        alert_output["urlhaus"]["url_blacklist_surbl"] = url_blacklist_surbl

        alert_output["urlhaus"]["url_tags"] = url_tags

    debug(alert_output)

    return(alert_output)

def send_event(msg, agent = None):

    if not agent or agent["id"] == "000":

        string = '1:urlhaus:{0}'.format(json.dumps(msg))

    else:

        string = '1:[{0}] ({1}) {2}->urlhaus:{3}'.format(agent["id"], agent["name"], agent["ip"] if "ip" in agent else "any", json.dumps(msg))

    debug(string)

    sock = socket(AF_UNIX, SOCK_DGRAM)

    sock.connect(socket_addr)

    sock.send(string.encode())

    sock.close()

if __name__ == "__main__":

    try:

        # Read arguments

        bad_arguments = False

        if len(sys.argv) >= 4:

            msg = '{0} {1} {2} {3} {4}'.format(now, sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] if len(sys.argv) > 4 else '')

            debug_enabled = (len(sys.argv) > 4 and sys.argv[4] == 'debug')

        else:

            msg = '{0} Wrong arguments'.format(now)

            bad_arguments = True

        # Logging the call

        f = open(log_file, 'a')

        f.write(msg +'\n')

        f.close()

        if bad_arguments:

            debug("# Exiting: Bad arguments.")

            sys.exit(1)

        # Main function

        main(sys.argv)

    except Exception as e:

        debug(str(e))

        raise

[Openime Oniagbi]

Okay. Let me test this and get back to you.

[Openime Oniagbi]

Hi Hakim,

Sorry for the late reply. I was able to reproduce the error you got and I have attached a file that fixes the problem.

The code is the same but the indentation is handled better.

Please test and let me know if it works for you also.

Regards,

[Hakim Mutalib]

Hi Openime,

sadly it still doesn't work, I'm not sure what the problem is, maybe it's on my end.

the url i use is : curl -s http://testmynids.org/uid/index.html 

The only thing appear in the wazuh dashboard is this : 



I am not sure why my Suricata doesn't detect the test url from your blog which is http://pastebin.com/raw/ZkwP7zPF

[Openime Oniagbi]

Hi,

I just tested it again and it ran successfully. See the screenshot below:



Please confirm that both your agent and Wazuh manager have internet access. Also, please use the last script I sent, restart your Wazuh manager and try again.

It should work now.

Regards,

[Hakim Mutalib]

I think it is not about the code but there is something wrong with my suricata.

It does not detect the ET Policy Curl alert. 

here is the log when I initiate the CURL request 

{"timestamp":"2022-08-18T08:50:50.202301+0000","flow_id":1970324793651718,"in_iface":"eth0","event_type":"http","src_ip":"2400:8901:0000:0000:f03c:93ff:feeb:81e9","src_port":57962,"dest_ip":"2606:4700:0010:0000:0000:0000:6814:448f","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"pastebin.com","url":"/raw/ZkwP7zPF","http_user_agent":"curl/7.68.0","http_method":"GET","protocol":"HTTP/1.1","status":301,"redirect":"https://pastebin.com/raw/ZkwP7zPF","length":3}}

FYI I am using ubuntu 20.04 and suricata v6.0.5

and here is the list of rules in /etc/suricata/rules




I have also attached my suricata config file. 

Thank you for your help Openime.

[Openime Oniagbi]

Hi Hakim,

I have sent a Suricata.rules file to you. I could not attach it here because the file is too large.

Follow the accompanying instructions and let me know if it works.

Regards,

On Thu, Aug 18, 2022 at 3:14 PM Openime Oniagbi <openime...@wazuh.com> wrote:

HI Hakim,

I have gone through your suricata.yaml file and your rule files are different from what I am using.

I have attached my rule file to this email. Please save to your rules directory and update your suricata.yaml file by adding the new rule file's name to it.

Restart the suricata service and test again.

Let me know if you have any questions or concerns.

Regards,

[Salwa Putri]

Hello! I have the same problem here with my URLHaus because it can't show the event when executing the curl pastebin command, can I get the suricata.yaml and config files to match?

Thanks in advance!