Custom integration issue

[M V]

Hello Wazuh-gurus,

I have a custom integration which has stopped working since upgrade to 4.10. I'm now on 4.14.1-1, and I am hoping for some guidance.

The integration (custom-homeassistant) was written following guidelines presented here. It essentially generates a custom payload to a web hook url. 

I'm attaching both the shell and python script herewith.  It loads correctly upon Wazuh-manager startup:

grep "integration" ../logs/ossec.log
2025/11/16 00:07:19 wazuh-integratord: INFO: Enabling integration for: 'custom-homeassistant'.
2025/11/16 00:18:53 wazuh-integratord: INFO: Enabling integration for: 'custom-homeassistant'.
2025/11/16 00:21:48 wazuh-integratord: INFO: Enabling integration for: 'custom-homeassistant'.
2025/11/16 01:28:43 wazuh-integratord: INFO: Enabling integration for: 'custom-homeassistant'.
2025/11/16 11:10:30 wazuh-integratord: INFO: Enabling integration for: 'custom-homeassistant'.

An alert which I expect to trigger the integration is based on a Headscale+Tailscale custom log processing upon a connection of a new node. Both the custom decoder and rules work as expected:

/var/ossec/integrations   v25.0.0$ ../bin/wazuh-logtest -d
2025-11-16 12:00:22,489 wazuh_logtest[INFO] Starting wazuh-logtest v4.14.1
2025-11-16 12:00:22,489 wazuh_logtest[INFO] Type one log per line

Nov 15 21:06:07 docker-worker-01 docker/headscale[1844836]: {"level":"info","caller":"/home/runner/work/headscale/headscale/hscontrol/poll.go:383","omitPeers":false,"stream":true,"node.id":6,"node.name":"niloufar-iphone","time":1763269567,"message":"node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0"}
2025-11-16 12:00:26,385 wazuh_logtest[INFO]
2025-11-16 12:00:26,385 wazuh_logtest[DEBUG] Request: {"version": 1, "origin": {"name": "wazuh-logtest", "module": "wazuh-logtest"}, "command": "log_processing", "parameters": {"location": "stdin", "log_format": "syslog", "event": "Nov 15 21:06:07 docker-worker-01 docker/headscale[1844836]: {\"level\":\"info\",\"caller\":\"/home/runner/work/headscale/headscale/hscontrol/poll.go:383\",\"omitPeers\":false,\"stream\":true,\"node.id\":6,\"node.name\":\"niloufar-iphone\",\"time\":1763269567,\"message\":\"node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0\"}"}}

2025-11-16 12:00:26,829 wazuh_logtest[DEBUG] Reply: {"error":0,"data":{"messages":["INFO: (7202): Session initialized with token 'fcad3b35'"],"token":"fcad3b35","output":{"timestamp":"2025-11-16T12:00:26.829-0800","rule":{"level":9,"description":"Headscale:: niloufar-iphone with NodeID = 6 connected to our private network !!!","id":"110611","firedtimes":1,"mail":false,"groups":["docker_headscale_logsinitial_access","tailscale","headscale","login"]},"agent":{"id":"000","name":"scanner.esco.ghaar"},"manager":{"name":"scanner.esco.ghaar"},"id":"1763323226.12289525","full_log":"Nov 15 21:06:07 docker-worker-01 docker/headscale[1844836]: {\"level\":\"info\",\"caller\":\"/home/runner/work/headscale/headscale/hscontrol/poll.go:383\",\"omitPeers\":false,\"stream\":true,\"node.id\":6,\"node.name\":\"niloufar-iphone\",\"time\":1763269567,\"message\":\"node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0\"}","predecoder":{"program_name":"docker/headscale","timestamp":"Nov 15 21:06:07","hostname":"docker-worker-01"},"decoder":{"parent":"docker_container_headscale","name":"docker_container_headscale"},"data":{"level":"info","caller":"/home/runner/work/headscale/headscale/hscontrol/poll.go:383","omitPeers":"false","stream":"true","node":{"id":"6","name":"niloufar-iphone"},"time":"1763269567","message":"node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0"},"location":"stdin"},"alert":true,"codemsg":0}}

2025-11-16 12:00:26,830 wazuh_logtest[DEBUG] {
  "messages": [
    "INFO: (7202): Session initialized with token 'fcad3b35'"
  ],
  "token": "fcad3b35",
  "output": {
    "timestamp": "2025-11-16T12:00:26.829-0800",
    "rule": {
      "level": 9,
      "description": "Headscale:: niloufar-iphone with NodeID = 6 connected to our private network !!!",
      "id": "110611",
      "firedtimes": 1,
      "mail": false,
      "groups": [
        "docker_headscale_logsinitial_access",
        "tailscale",
        "headscale",
        "login"
      ]
    },
    "agent": {
      "id": "000",
      "name": "scanner.esco.ghaar"
    },
    "manager": {
      "name": "scanner.esco.ghaar"
    },
    "id": "1763323226.12289525",
    "full_log": "Nov 15 21:06:07 docker-worker-01 docker/headscale[1844836]: {\"level\":\"info\",\"caller\":\"/home/runner/work/headscale/headscale/hscontrol/poll.go:383\",\"omitPeers\":false,\"stream\":true,\"node.id\":6,\"node.name\":\"niloufar-iphone\",\"time\":1763269567,\"message\":\"node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0\"}",
    "predecoder": {
      "program_name": "docker/headscale",
      "timestamp": "Nov 15 21:06:07",
      "hostname": "docker-worker-01"
    },
    "decoder": {
      "parent": "docker_container_headscale",
      "name": "docker_container_headscale"
    },
    "data": {
      "level": "info",
      "caller": "/home/runner/work/headscale/headscale/hscontrol/poll.go:383",
      "omitPeers": "false",
      "stream": "true",
      "node": {
        "id": "6",
        "name": "niloufar-iphone"
      },
      "time": "1763269567",
      "message": "node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0"
    },
    "location": "stdin"
  },
  "alert": true,
  "codemsg": 0
}
2025-11-16 12:00:26,830 wazuh_logtest[INFO] **Phase 1: Completed pre-decoding.
2025-11-16 12:00:26,830 wazuh_logtest[INFO] full event: 'Nov 15 21:06:07 docker-worker-01 docker/headscale[1844836]: {"level":"info","caller":"/home/runner/work/headscale/headscale/hscontrol/poll.go:383","omitPeers":false,"stream":true,"node.id":6,"node.name":"niloufar-iphone","time":1763269567,"message":"node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0"}'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] timestamp: 'Nov 15 21:06:07'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] hostname: 'docker-worker-01'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] program_name: 'docker/headscale'
2025-11-16 12:00:26,830 wazuh_logtest[INFO]
2025-11-16 12:00:26,830 wazuh_logtest[INFO] **Phase 2: Completed decoding.
2025-11-16 12:00:26,830 wazuh_logtest[INFO] name: 'docker_container_headscale'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] parent: 'docker_container_headscale'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] caller: '/home/runner/work/headscale/headscale/hscontrol/poll.go:383'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] level: 'info'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] message: 'node has connected, mapSession: 0xc0000b0600, chan: 0xc00058acb0'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] node.id: '6'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] node.name: 'niloufar-iphone'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] omitPeers: 'false'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] stream: 'true'
2025-11-16 12:00:26,830 wazuh_logtest[INFO] time: '1763269567'
2025-11-16 12:00:26,830 wazuh_logtest[INFO]
2025-11-16 12:00:26,830 wazuh_logtest[INFO] **Phase 3: Completed filtering (rules).
2025-11-16 12:00:26,830 wazuh_logtest[INFO] id: '110611'
2025-11-16 12:00:26,831 wazuh_logtest[INFO] level: '9'
2025-11-16 12:00:26,831 wazuh_logtest[INFO] description: 'Headscale:: niloufar-iphone with NodeID = 6 connected to our private network !!!'
2025-11-16 12:00:26,831 wazuh_logtest[INFO] groups: '['docker_headscale_logsinitial_access', 'tailscale', 'headscale', 'login']'
2025-11-16 12:00:26,831 wazuh_logtest[INFO] firedtimes: '1'
2025-11-16 12:00:26,831 wazuh_logtest[INFO] mail: 'False'
2025-11-16 12:00:26,831 wazuh_logtest[INFO] **Alert to be generated.

The ossec.conf has the above rule.id (110611) which should, in theory, trigger the integration. 

<integration>
    <name>custom-homeassistant</name>
    <hook_url>https://homeassistant.esco.ghaar/api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i</hook_url>
    <level>9</level>
    <!--<group>authentication_failed,invalid_login,syslog,sshd,ubuntu22,ubuntu24,ubuntu_22_caddy,caddy_sarbi_website,caddy_fw,ubuntu_24_caddy,freebsd,mongodb,pihole,postgresql,rancher,</group>-->
    <group>authentication_failed,invalid_login,shutdown,system_shutdown,restart</group>
    <alert_format>json</alert_format>
    <rule_id>504,505,560,2945,5103,5701,5702,5703,5704,5705,5710,5712,5714,5716,5718,5719,5720,5748,5758,5760,5763,12101,31103,31104,31105,31152,31153,31154,31164,31165,31508,31511,31533,100002,100004,100005,100006,100007,100008,100009,110013,110014,110212,110610,110611</rule_id>
    <timeout>30</timeout>
    <retries>2</retries>
  </integration> 

Everything should be in order yet I do not see a HTTP request come out of Wazuh-manager to the endpoint. 

If I try to manually trigger the integration (by storing just the single alert in a temporary file), then it works. So, I'm out of ideas as to how to further troubleshoot it. Help please

-- manual trigger--

The above alert when stored in /tmp/test_alert.json :

{"timestamp":"2025-11-16T10:41:03.799-0800","rule":{"level":9,"description":"Headscale:: localhost with NodeID = 7 connected to our private network !!!","id":"110611","firedtimes":7,"mail":false,"groups":["docker_headscale_logsinitial_access","tailscale","headscale","login"]},"agent":{"id":"040","name":"docker-worker-01.esco.ghaar","ip":"192.168.100.24"},"manager":{"name":"scanner.esco.ghaar"},"id":"1763318463.3217437","full_log":"Nov 16 10:41:03 docker-worker-01 docker/headscale[1844836]: {\"level\":\"info\",\"node.id\":7,\"node.name\":\"localhost\",\"time\":1763318463,\"message\":\"Node connected\"}","predecoder":{"program_name":"docker/headscale","timestamp":"Nov 16 10:41:03","hostname":"docker-worker-01"},"decoder":{"parent":"docker_container_headscale","name":"docker_container_headscale"},"data":{"level":"info","node":{"id":"7","name":"localhost"},"time":"1763318463","message":"Node connected"},"location":"/var/log/docker/headscale.log"}

and triggered like so:

/var/ossec/integrations   v25.0.0$ ../framework/python/bin/python3 custom-homeassistant.py /tmp/test_alert.json "" "https://homeassistant.esco.ghaar/api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i"
[DEBUG] 2025-11-16 12:12:58,923 [<module>::178]--> Inputs:: Sun Nov 16 12:12:58 PST 2025 /tmp/test_alert.json  https://homeassistant.esco.ghaar/api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i
[DEBUG] 2025-11-16 12:12:58,923 [main::38]--> Alert file location: /tmp/test_alert.json
[DEBUG] 2025-11-16 12:12:58,923 [main::39]--> User:
[DEBUG] 2025-11-16 12:12:58,923 [main::40]--> API Key:
[DEBUG] 2025-11-16 12:12:58,923 [main::41]--> Webhook URL: https://homeassistant.esco.ghaar/api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i
[DEBUG] 2025-11-16 12:12:58,923 [main::114]--> {
 "message": {
  "type": "Wazuh_Alert",
  "alertlevel": "9",
  "ruleid": "110611",
  "vm": "docker-worker-01.esco.ghaar",
  "agentid": "040",
  "details": "Headscale:: localhost with NodeID = 7 connected to our private network !!!",
  "fulllog": "Nov 16 10:41:03 docker-worker-01 docker/headscale[1844836]: {\"level\":\"info\",\"node.id\":7,\"node.name\":\"localhost\",\"time\":1763318463,\"message\":\"Node connected\"}"
 }
}
[DEBUG] 2025-11-16 12:12:58,924 [main::117]--> Preparing HTTP POST to webhook ...
[DEBUG] 2025-11-16 12:12:58,924 [main::127]--> POST /api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i HTTP/1.1
Content-Type: application/json
Content-Length: 407

{"message": {"type": "Wazuh_Alert", "alertlevel": "9", "ruleid": "110611", "vm": "docker-worker-01.esco.ghaar", "agentid": "040", "details": "Headscale:: localhost with NodeID = 7 connected to our private network !!!", "fulllog": "Nov 16 10:41:03 docker-worker-01 docker/headscale[1844836]: {\"level\":\"info\",\"node.id\":7,\"node.name\":\"localhost\",\"time\":1763318463,\"message\":\"Node connected\"}"}}
[DEBUG] 2025-11-16 12:12:58,924 [main::129]--> Sending HTTP POST to webhook ...
[DEBUG] 2025-11-16 12:12:58,924 [_new_conn::1049]--> Starting new HTTPS connection (1): homeassistant.esco.ghaar:443
/var/ossec/framework/python/lib/python3.10/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'homeassistant.esco.ghaar'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
[DEBUG] 2025-11-16 12:12:58,933 [_make_request::544]--> https://homeassistant.esco.ghaar:443 "POST /api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i HTTP/1.1" 200 0
[DEBUG] 2025-11-16 12:12:58,933 [main::134]--> Response: <Response [200]>
[DEBUG] 2025-11-16 12:12:58,933 [main::135]--> {'Server': 'nginx', 'Date': 'Sun, 16 Nov 2025 20:12:58 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Referrer-Policy': 'no-referrer', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'}

--manual trigger-

-----

Although not directly relevant, here's the custom decoder and rule file for reference. 

--Decoder--

<decoder name="docker_container_headscale">
    <program_name type="pcre2">^docker\/headscale</program_name>
    <!--<prematch type="pcre2">docker\/</prematch> -->
</decoder>

<decoder name="docker_container_headscale_json">
    <parent>docker_container_headscale</parent>
    <prematch type="pcre2">\s*</prematch>
    <plugin_decoder offset="after_prematch">JSON_Decoder</plugin_decoder>
</decoder>

--Rule--

<group name="docker_headscale_logs">

    <!--  Docker container Headscale Rule Base  -->
    <rule id="110600" level="0">
     <decoded_as>docker_container_headscale</decoded_as>
      <description>Processing headscale container log</description>
    </rule>

    <rule id="110610" level="9">
        <if_sid>110600</if_sid>
        <field name="message" type="pcre2">(?i)diconnected</field>
        <description>Headscale:: $(node.name) with NodeID = $(node.id) disconnected from our private network !!!</description>
        <group>initial_access,tailscale,headscale,login</group>
    </rule>

    <rule id="110611" level="9">
        <if_sid>110600</if_sid>
        <field name="message" type="pcre2">(?i)connected</field>
        <description>Headscale:: $(node.name) with NodeID = $(node.id) connected to our private network !!!</description>
        <group>initial_access,tailscale,headscale,login</group>
    </rule>

</group>

[Karlo Balmores Veranga]

Hi,

This behavior is expected after the upgrade. Starting with Wazuh 4.7, the <alert_format>json</alert_format> option became mandatory for all integrations, including custom ones. If this tag is missing, the integration will load but will not be triggered automatically.

You can find the full explanation and configuration requirements here:
https://documentation.wazuh.com/current/user-manual/manager/integration-with-external-apis.html#configuration-options

Please add the missing tag to your integration block in ossec.conf and restart the Wazuh Manager. After that, the integration should work as expected.

Let us know if you need any further help.

[M V]

Hello Karlo,

Thank you so much for spending the time to review my logs, and to provide a suggested way forward.

Per the below mentioned snippet from server's ossec.conf, the alert type was already configured to be json.

  <integration>
    <name>custom-homeassistant</name>
    <hook_url>https://homeassistant.esco.ghaar/api/webhook/webhooks-EAuQtPawfQ6aT5bp-yP23p5i</hook_url>
    <level>9</level>
    <!--<group>authentication_failed,invalid_login,syslog,sshd,ubuntu22,ubuntu24,ubuntu_22_caddy,caddy_sarbi_website,caddy_fw,ubuntu_24_caddy,freebsd,mongodb,pihole,postgresql,rancher,</group>-->
    <group>authentication_failed,invalid_login,shutdown,system_shutdown,restart</group>
    <alert_format>json</alert_format>
<rule_id>504,505,560,2945,5103,5701,5702,5703,5704,5705,5710,5712,5714,5716,5718,5719,5720,5748,5758,5760,5763,12101,31103,31104,31105,31152,31153,31154,31164,31165,31508,31511,31533,100002,100004,100005,100006,100007,100008,100009,110013,110014,110212,110610,110611</rule_id>
    <timeout>30</timeout>
    <retries>2</retries>
  </integration>

/r

maulik

[Karlo Balmores Veranga]

Hi,

 You already have <alert_format>json</alert_format>, so we need to check if alerts are matching your filters and whether the integration script is being executed. Please confirm script location/permissions and check integrations.log for trigger errors.”  

[M V]

Thank you again, Karlo.

Per the logs mentioned in my original email, no errors appear in integration.log. Alerts match when triggered manually via logtest. Permissions look consistent (please see below) as well

ll
total 100K
drwxr-x---  2 root wazuh 4.0K Nov 17 14:08 ./
drwxr-x--- 20 root wazuh 4.0K Nov 15 17:42 ../
-rwxr-x---  1 root wazuh 1.1K Nov 16 11:32 custom-homeassistant*
-rwxr-x---  1 root wazuh 6.2K Nov 17 14:08 custom-homeassistant.py*
-rwxr-x---  1 root wazuh 1.1K Nov  8 04:00 maltiverse*
-rwxr-x---  1 root wazuh  21K Nov  8 04:00 maltiverse.py*
-rwxr-x---  1 root wazuh 1.1K Nov  8 04:00 pagerduty*
-rwxr-x---  1 root wazuh 6.3K Nov  8 04:00 pagerduty.py*
-rwxr-x---  1 root wazuh 1.1K Nov  8 04:00 shuffle*
-rwxr-x---  1 root wazuh 7.1K Nov  8 04:00 shuffle.py*
-rwxr-x---  1 root wazuh 1.1K Nov  8 04:00 slack*
-rwxr-x---  1 root wazuh 6.7K Nov  8 04:00 slack.py*
-rwxr-x---  1 root wazuh 1.1K Nov  8 04:00 virustotal*
-rwxr-x---  1 root wazuh  11K Nov  8 04:00 virustotal.py*

.

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/Zus_0BOzPYg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wazuh/5fc836d3-b327-42f2-b723-c8bceca7a722n%40googlegroups.com.

[Karlo Balmores Veranga]

Hi,
The integration is not triggering because the alert does not match your integration filters, not because the script is broken. Your rule 110611 fires correctly, but its groups (initial_access, tailscale, headscale, login) do not match the <group> values configured in the integration (authentication_failed, shutdown, restart, etc.).
Manual execution works because it bypasses filtering.

Fix by either:

• Adding matching groups to the <group> field, or

• Removing <group> entirely and relying on <rule_id> only

Once the alert matches the filters, the integration will trigger normally.

[M V]

Karlo,

I couldn’t thank you enough. Removing <groups> resolved the issue. Perhaps you could consider reflecting the same in the next revision of associated docs. To me, it wasn’t clear whether and which elements in <integration> block had mutual exclusivity and preference. 

Much obliged.