We replayed 24 ATT&CK techniques against a default Wazuh install. Three of them raised an alert.

[Nguyen Xuan Dong]

Not three percent. Three techniques. Denominator 24, numerator 3.

Default install: stock rules, no Sysmon, no ScriptBlock logging, no tuning. A technique counted as detected only if it produced an alert a human would actually see.

The four chains you would most want to catch — discovery, collection, exfiltration, C2 — produced zero alerts between them.

The one technique we ran end to end on this platform is T1110.001, SSH brute force, 20+ failed logins against Wazuh 4.14.2 with a custom rule. It alerted twice, MTTD 25s and 45s.

Dates, because they matter: the Wazuh 4.14.2 brute-force run is 15 June 2026. The 24-technique replay comes from our published DRI Vol.1 and we are not putting a run date on it here, because we cannot point you at one. Neither has been re-run since. If the default rule set has changed in the meantime our number is stale and we would like to know.

Full method, the negative control that makes the other rows mean anything, and a list of what we did not measure: https://github.com/xuxu298/siem-replay-24-techniques

Corrections wanted. If you think the 3/24 is wrong, tell us which technique should have fired and on what rule. We will re-run it and publish the correction with the same prominence as the original claim.

[Matías Exequiel García]

Hi Nguyen, 

Thanks for sharing the methodology and for being open to feedback.

There are a couple of points that would be helpful to clarify before interpreting the 3/24 result as representative of a default Wazuh installation.

First, could you share the Wazuh version, endpoint OS, exact ossec.conf / agent.conf, and the custom/community rules used for the 24-technique replay? I saw that Wazuh 4.14.2 is mentioned for the separate SSH brute-force test, but it is not clear whether the 3/24 replay was also performed on that version.

One important detail is the 15–25 second observation window. By default, Wazuh FIM monitors paths such as /etc, /usr/bin, /usr/sbin, /bin, /sbin, and /boot, but the default scan interval is 12 hours and real-time monitoring is not enabled by default.

Because of that, a file modification under /etc, for example, could be marked as MISSED after 25 seconds even though the default FIM configuration would detect it on the next scheduled scan. Wazuh already includes built-in rules for file creation, modification, and deletion.

It would therefore be useful to have the exact commands and paths used for the file-related techniques. Modifying /etc/sudoers, for example, is quite different from modifying ~/.ssh/authorized_keys, since the latter is not monitored by default.

I would also be interested in the T1110.001 result. Wazuh includes built-in SSH brute-force correlation rules, so if 20+ failed logins only produced rule 5503 and a custom rule, seeing the raw SSH/PAM events and the exact test command would help understand why the built-in correlation did not match.

It may also be useful to distinguish between:

1. the required telemetry not being collected by default;
2. the telemetry being collected but no rule matching;
3. the detection existing but not firing within the selected observation window.

Those cases are quite different, and separating them would make the results much easier to interpret and reproduce.

If you can share the per-technique commands and configuration, I’d be happy to look at the MISSED cases in more detail.

[Nguyen Xuan Dong]

Matías — you are right about the window, and it costs us four rows.

Default FIM in that build scans on a 12-hour interval with real-time monitoring off. A 15-25 second query window cannot see a scheduled scan, so the four file-based persistence techniques we ran — cron (T1053.003), sudoers (T1548.003), authorized_keys (T1098.004) and permission change (T1222.002) — were never observable inside the window we chose. Those four are not evidence that a rule is missing. They are evidence that our window was shorter than the default scan interval. T1098.004 is a separate case again: as you say, ~/.ssh/authorized_keys is not in the default monitored set at all, so that one is scope rather than timing.

Your three-way split is the right instrument and we did not have it. Our published notes separate "telemetry not collected" from "collected but no rule matched". They do not separate your third case, the detection existing and not firing inside the observation window. The four rows above are exactly that third case, and they were being read as the second.

What I cannot give you is the thing you asked for first. For the 24-technique replay I cannot point at a source file holding the Wazuh version, the endpoint OS, the ossec.conf and agent.conf, or the per-technique commands. 4.14.2 belongs to the T1110.001 run of 15 June 2026, and I should not let that version attach itself to the replay. Until that record is attached, the 3-of-24 is not reproducible from anything we have published, and I am treating it that way rather than letting the number stand on its own.

Same answer on T1110.001. What is written down is default rule 5503 plus custom rule 100113. I do not have the raw SSH and PAM events, so I cannot tell you why the built-in brute-force correlation is absent from that record — whether it fired and was not written down, or did not fire. I am not going to guess in your direction or in ours.

One thing that would help. Of the 21 silent techniques, which would you expect a stock build to catch? That is the list I would re-run first.

[Matías Exequiel García]

Thanks, I appreciate the detailed response.

For a first re-run, I would keep the list fairly small rather than claiming coverage for techniques where the default agent does not collect the required telemetry.

The clearest candidates to me are the FIM cases that touch paths monitored by default:

• T1053.003 (Cron): if the test creates or modifies a file under /etc/cron*, I would expect the standard FIM rules to report it on the next scheduled scan (554 for a new file or 550 for a modification).

• T1548.003 (Sudoers): changes to /etc/sudoers or /etc/sudoers.d/* should similarly be visible through FIM, normally as rule 550.

• T1222.002 (Permission change): if the permission change is made to a file in one of the default monitored paths, I would expect rule 550, since permissions are checked by default.

• T1070.004 (File deletion): I would also re-check this one if the deleted file was inside a default-monitored path. In that case I would expect the standard file deletion rule (553). If the test used /tmp, a home directory, etc., then I would classify it as a scope/telemetry gap instead.

I would not make the same claim for most of the discovery, credential-read, collection, exfiltration or C2 cases. Without Auditd/command auditing or network telemetry, I would expect many of those to fall into the “telemetry not collected” category rather than “a stock rule should have fired.”

So I’d probably start with those four, using a window long enough to include the scheduled FIM scan, and document the exact path/action for each one. That should give us a much cleaner baseline.

[Nguyen Xuan Dong]

Matías — taking your four and dropping the rest of the list is the right scope, and it is

narrower than the one we would have drawn ourselves. Agreed on all four.

One of them we can answer without re-running anything, and the answer does not go our way. For

the file deletion case we did not record which path the deleted file sat in. So we cannot tell

you today whether it belongs with the file integrity monitoring rows or in the scope gap, and we

are not going to guess it into whichever column reads better. It stays unclassified until a

re-run says otherwise.

The other three we will re-run with the observation window set to at least one full scheduled

scan interval on that build, and publish, per technique: the exact path, the exact action, the

interval we used, and what came back — including the rows where your expectation holds and ours

does not.

One thing we would rather you decide than us: which version the re-run should be on. You

were right that no version is published in that file, and the fix for that is a version you

picked rather than one you have to take on trust from us.

We are not going to promise you a delivery date for the re-run itself, because we do not control

whether the lab comes back clean. What we will commit to is a status: we will post in this thread

by 10 September 2026 saying where the re-run stands — and if the status on that date is that we

could not run it, or that it slipped, that is what the post will say. A commitment with no date

attached is one that dies quietly, and this thread deserves better than that.

The correction is now at the top of the repo README instead of the bottom, with the original

sentence left standing beside it rather than quietly edited away. That was your reading of the

file, not ours.

https://github.com/xuxu298/siem-replay-24-techniques

[Matías Exequiel García]

Hi Nguyen,

I would suggest using Wazuh 4.14.7 for the re-run, since it is the latest stable version currently available. I would use 4.14.7 for both the manager and the agent.