Can we parse JSON string

[Dhiraj Ambigapathi]

I was wondering if we can parse a JSON string. My reverse proxy docker creates a JSON string log file, which can be converted to JSON using a python script. I was wondering if Wazuh agent has some configuration to do this or do I need to write a regex or any alternate solution.

JSON string Log
{"log":"{\"transaction\":{\"client_ip\":\"187.145.96.30\",\"time_stamp\":\"Fri Jul 28 12:54:08 2023\",\"server_id\":\"08009b29418b10ad42d8757275cce1b4df950f9a\",\"client_port\":32791,\"host_ip\":\"192.168.0.6\",\"host_port\":80,\"unique_id\":\"169054884881.853196\",\"request\":{\"method\":\"GET\",\"http_version\":1.1,\"uri\":\"/\",\"headers\":{\"Host\":\"18.235.43.100:80\",\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36\",\"Content-Length\":\"0\"}},\"response\":{\"body\":\"\u003c!DOCTYPE html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\\n\u003chead\u003e\\n  \u003cmeta charset=\\\"UTF-8\\\" /\u003e\\n  \u003clink rel=\\\"icon\\\" href=\\\"/favicon.svg\\\" type=\\\"image/svg+xml\\\" /\u003e\\n  \u003cmeta name=\\\"viewport\\\" content=\\\"width=device-width, initial-scale=1.0\\\" /\u003e\\n  \u003cscript src=\\\"./ckeditor/ckeditor.js\\\"\u003e\u003c/script\u003e\\n  \u003cscript type=\\\"module\\\" crossorigin src=\\\"/assets/index.18298ef0.js\\\"\u003e\u003c/script\u003e\\n  \u003clink rel=\\\"stylesheet\\\" href=\\\"/assets/index.76e4662b.css\\\"\u003e\\n\u003c/head\u003e\\n\\n\u003cbody\u003e\\n  \u003cdiv id=\\\"app\\\"\u003e\u003c/div\u003e\\n  \\n\u003c/body\u003e\\n\\n\u003c/html\u003e\\n\",\"http_code\":200,\"headers\":{\"Server\":\"nginx\",\"Date\":\"Fri, 28 Jul 2023 12:54:08 GMT\",\"Content-Length\":\"452\",\"Content-Type\":\"text/html\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Connection\":\"keep-alive\",\"ETag\":\"\\\"64b7baef-1c4\\\"\",\"Accept-Ranges\":\"bytes\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.8 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"Enabled\",\"components\":[\"OWASP_CRS/3.3.4\\\"\"]},\"messages\":[{\"message\":\"Host header is a numeric IP address\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `^[\\\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `18.235.43.100:80' )\",\"reference\":\"o0,16v21,16\",\"ruleId\":\"920350\",\"file\":\"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\",\"lineNumber\":\"719\",\"data\":\"18.235.43.100:80\",\"severity\":\"4\",\"ver\":\"OWASP_CRS/3.3.4\",\"rev\":\"\",\"tags\":[\"modsecurity\",\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-protocol\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/210/272\",\"PCI/6.5.10\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}\n","stream":"stdout","time":"2023-07-28T12:54:08.174449533Z"}


So I did write a regex too, but it is not working for my decoder, even though it works for regex101.
Decoder
<decoder name="ax">
    <prematch type="pcre2">^\{"log":</prematch>
    <regex offset="after_prematch" type="pcre2">{"log":"(\d+.\d+.\d+.\d+).-.-.\[(\d+\/\w+\/\w+):(\d+:\d+:\d+)\s.....\]\s."(GET|POST)\s(\s+|\S+)\s(HTTP\/\d+..)\\"\s(\d+)\s(\d+)\s........([a-zA-Z0-9. \/\(;_\),]+)\s([a-zA-Z0-9. \/\\]+)[\s+|\S+]+</regex>
    <order>attackerip,date,time,HTTP_Method,URI,HTTP_Protocol,HTTP_Status,Response_Length,Browser</order>
</decoder>

log
{"log":"135.125.246.189 - - [31/Jul/2023:09:29:08 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.
0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-31T09:29:08.855788269Z"}

Regex101


Apologies to combining two queries in 1 conversation.

[Luis González Romero]

Hello Dhiraj Ambigapathi, hope you’re great.

Here you have some thing you have to think about. You have a JSON that can be decoded using the Wazuh’s json decoder but the log field contains a JSON string value.

You can follow these approaches:

1. Formatting your log as a JSON instead, so they can be fully collected without effort
2. If you want to use a JSON string format, you should not use a JSON to store them. You won’t be able to configure the JSON decoder. So if you have a log that is not a JSON you can create your own decoders

Hope this helps you,
Luis

​

[Dhiraj Ambigapathi]

Hi Luis,

I'm doing well,
Hope you're doing good.
I've couple queries regarding this
Can I use Wazuh agent to run a python script to convert json string file to json file to send to wazuh manager or Can I pipe a command like "Docker logs container" output to wazuh agent, because the output is JSON. I've written a parser but its not working as per my previous message.

[Luis González Romero]

The problem here is that JSON decoder can’t be customized. So you can’t write a parser using json format. If your log would be a JSON it would perfectly fit, but in your case the log field is not json

root@manager44:/home/vagrant# /var/ossec/bin/wazuh-logtest Starting wazuh-logtest v4.4.4 Type one log per line {"log":"{\"transaction\":{\"client_ip\":\"187.145.96.30\",\"time_stamp\":\"Fri Jul 28 12:54:08 2023\",\"server_id\":\"08009b29418b10ad42d8757275cce1b4df950f9a\",\"client_port\":32791,\"host_ip\":\"192.168.0.6\",\"host_port\":80,\"unique_id\":\"169054884881.853196\",\"request\":{\"method\":\"GET\",\"http_version\":1.1,\"uri\":\"/\",\"headers\":{\"Host\":\"18.235.43.100:80\",\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36\",\"Content-Length\":\"0\"}},\"response\":{\"body\":\"\u003c!DOCTYPE html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\\n\u003chead\u003e\\n \u003cmeta charset=\\\"UTF-8\\\" /\u003e\\n \u003clink rel=\\\"icon\\\" href=\\\"/favicon.svg\\\" type=\\\"image/svg+xml\\\" /\u003e\\n \u003cmeta name=\\\"viewport\\\" content=\\\"width=device-width, initial-scale=1.0\\\" /\u003e\\n \u003cscript src=\\\"./ckeditor/ckeditor.js\\\"\u003e\u003c/script\u003e\\n \u003cscript type=\\\"module\\\" crossorigin src=\\\"/assets/index.18298ef0.js\\\"\u003e\u003c/script\u003e\\n \u003clink rel=\\\"stylesheet\\\" href=\\\"/assets/index.76e4662b.css\\\"\u003e\\n\u003c/head\u003e\\n\\n\u003cbody\u003e\\n \u003cdiv id=\\\"app\\\"\u003e\u003c/div\u003e\\n \\n\u003c/body\u003e\\n\\n\u003c/html\u003e\\n\",\"http_code\":200,\"headers\":{\"Server\":\"nginx\",\"Date\":\"Fri, 28 Jul 2023 12:54:08 GMT\",\"Content-Length\":\"452\",\"Content-Type\":\"text/html\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Connection\":\"keep-alive\",\"ETag\":\"\\\"64b7baef-1c4\\\"\",\"Accept-Ranges\":\"bytes\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.8 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"Enabled\",\"components\":[\"OWASP_CRS/3.3.4\\\"\"]},\"messages\":[{\"message\":\"Host header is a numeric IP address\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `^[\\\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `18.235.43.100:80' )\",\"reference\":\"o0,16v21,16\",\"ruleId\":\"920350\",\"file\":\"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\",\"lineNumber\":\"719\",\"data\":\"18.235.43.100:80\",\"severity\":\"4\",\"ver\":\"OWASP_CRS/3.3.4\",\"rev\":\"\",\"tags\":[\"modsecurity\",\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-protocol\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/210/272\",\"PCI/6.5.10\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}\n","stream":"stdout","time":"2023-07-28T12:54:08.174449533Z"} **Phase 1: Completed pre-decoding. full event: '{"log":"{\"transaction\":{\"client_ip\":\"187.145.96.30\",\"time_stamp\":\"Fri Jul 28 12:54:08 2023\",\"server_id\":\"08009b29418b10ad42d8757275cce1b4df950f9a\",\"client_port\":32791,\"host_ip\":\"192.168.0.6\",\"host_port\":80,\"unique_id\":\"169054884881.853196\",\"request\":{\"method\":\"GET\",\"http_version\":1.1,\"uri\":\"/\",\"headers\":{\"Host\":\"18.235.43.100:80\",\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36\",\"Content-Length\":\"0\"}},\"response\":{\"body\":\"\u003c!DOCTYPE html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\\n\u003chead\u003e\\n \u003cmeta charset=\\\"UTF-8\\\" /\u003e\\n \u003clink rel=\\\"icon\\\" href=\\\"/favicon.svg\\\" type=\\\"image/svg+xml\\\" /\u003e\\n \u003cmeta name=\\\"viewport\\\" content=\\\"width=device-width, initial-scale=1.0\\\" /\u003e\\n \u003cscript src=\\\"./ckeditor/ckeditor.js\\\"\u003e\u003c/script\u003e\\n \u003cscript type=\\\"module\\\" crossorigin src=\\\"/assets/index.18298ef0.js\\\"\u003e\u003c/script\u003e\\n \u003clink rel=\\\"stylesheet\\\" href=\\\"/assets/index.76e4662b.css\\\"\u003e\\n\u003c/head\u003e\\n\\n\u003cbody\u003e\\n \u003cdiv id=\\\"app\\\"\u003e\u003c/div\u003e\\n \\n\u003c/body\u003e\\n\\n\u003c/html\u003e\\n\",\"http_code\":200,\"headers\":{\"Server\":\"nginx\",\"Date\":\"Fri, 28 Jul 2023 12:54:08 GMT\",\"Content-Length\":\"452\",\"Content-Type\":\"text/html\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Connection\":\"keep-alive\",\"ETag\":\"\\\"64b7baef-1c4\\\"\",\"Accept-Ranges\":\"bytes\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.8 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"Enabled\",\"components\":[\"OWASP_CRS/3.3.4\\\"\"]},\"messages\":[{\"message\":\"Host header is a numeric IP address\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `^[\\\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `18.235.43.100:80' )\",\"reference\":\"o0,16v21,16\",\"ruleId\":\"920350\",\"file\":\"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\",\"lineNumber\":\"719\",\"data\":\"18.235.43.100:80\",\"severity\":\"4\",\"ver\":\"OWASP_CRS/3.3.4\",\"rev\":\"\",\"tags\":[\"modsecurity\",\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-protocol\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/210/272\",\"PCI/6.5.10\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}\n","stream":"stdout","time":"2023-07-28T12:54:08.174449533Z"}' **Phase 2: Completed decoding. name: 'json' log: '{"transaction":{"client_ip":"187.145.96.30","time_stamp":"Fri Jul 28 12:54:08 2023","server_id":"08009b29418b10ad42d8757275cce1b4df950f9a","client_port":32791,"host_ip":"192.168.0.6","host_port":80,"unique_id":"169054884881.853196","request":{"method":"GET","http_version":1.1,"uri":"/","headers":{"Host":"18.235.43.100:80","User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36","Content-Length":"0"}},"response":{"body":"<!DOCTYPE html>\n<html lang=\"en\">\n\n<head>\n <meta charset=\"UTF-8\" />\n <link rel=\"icon\" href=\"/favicon.svg\" type=\"image/svg+xml\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\" />\n <script src=\"./ckeditor/ckeditor.js\"></script>\n <script type=\"module\" crossorigin src=\"/assets/index.18298ef0.js\"></script>\n <link rel=\"stylesheet\" href=\"/assets/index.76e4662b.css\">\n</head>\n\n<body>\n <div id=\"app\"></div>\n \n</body>\n\n</html>\n","http_code":200,"headers":{"Server":"nginx","Date":"Fri, 28 Jul 2023 12:54:08 GMT","Content-Length":"452","Content-Type":"text/html","Last-Modified":"Wed, 19 Jul 2023 10:29:03 GMT","Last-Modified":"Wed, 19 Jul 2023 10:29:03 GMT","Connection":"keep-alive","ETag":"\"64b7baef-1c4\"","Accept-Ranges":"bytes"}},"producer":{"modsecurity":"ModSecurity v3.0.8 (Linux)","connector":"ModSecurity-nginx v1.0.3","secrules_engine":"Enabled","components":["OWASP_CRS/3.3.4\""]},"messages":[{"message":"Host header is a numeric IP address","details":{"match":"Matched \"Operator `Rx' with parameter `^[\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `18.235.43.100:80' )","reference":"o0,16v21,16","ruleId":"920350","file":"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf","lineNumber":"719","data":"18.235.43.100:80","severity":"4","ver":"OWASP_CRS/3.3.4","rev":"","tags":["modsecurity","application-multi","language-multi","platform-multi","attack-protocol","paranoia-level/1","OWASP_CRS","capec/1000/210/272","PCI/6.5.10"],"maturity":"0","accuracy":"0"}}]}} ' stream: 'stdout' time: '2023-07-28T12:54:08.174449533Z' **Phase 3: Completed filtering (rules). id: '1002' level: '2' description: 'Unknown problem somewhere in the system.' groups: '['syslog', 'errors']' firedtimes: '1' gpg13: '['4.3']' mail: 'False' ​

As you can see, the log field is not collected as expected. You can notice that the JSON special characters are escaped like \”.

If the log field were a JSON, it would be decoded like this:

**Phase 2: Completed decoding. name: 'json' transaction.client_ip: '187.145.96.30' transaction.client_port: '32791' transaction.host_ip: '192.168.0.6' transaction.host_port: '80' transaction.messages: '[{'message': 'Host header is a numeric IP address', 'details': {'match': 'Matched "Operator `Rx\' with parameter `^[\\d.:]+$\' against variable `REQUEST_HEADERS:Host\' (Value: `18.235.43.100:80\' )', 'reference': 'o0,16v21,16', 'ruleId': '920350', 'file': '/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf', 'lineNumber': '719', 'data': '18.235.43.100:80', 'severity': '4', 'ver': 'OWASP_CRS/3.3.4', 'rev': '', 'tags': ['modsecurity', 'application-multi', 'language-multi', 'platform-multi', 'attack-protocol', 'paranoia-level/1', 'OWASP_CRS', 'capec/1000/210/272', 'PCI/6.5.10'], 'maturity': '0', 'accuracy': '0'}}]' transaction.producer.components: '['OWASP_CRS/3.3.4"']' transaction.producer.connector: 'ModSecurity-nginx v1.0.3' transaction.producer.modsecurity: 'ModSecurity v3.0.8 (Linux)' transaction.producer.secrules_engine: 'Enabled' transaction.request.headers.Content-Length: '0' transaction.request.headers.Host: '18.235.43.100:80' transaction.request.headers.User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36' transaction.request.http_version: '1.100000' transaction.request.method: 'GET' transaction.request.uri: '/' transaction.response.body: '<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8" /> <link rel="icon" href="/favicon.svg" type="image/svg+xml" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <script src="./ckeditor/ckeditor.js"></script> <script type="module" crossorigin src="/assets/index.18298ef0.js"></script> <link rel="stylesheet" href="/assets/index.76e4662b.css"> </head> <body> <div id="app"></div> </body> </html> ' transaction.response.headers.Accept-Ranges: 'bytes' transaction.response.headers.Connection: 'keep-alive' transaction.response.headers.Content-Length: '452' transaction.response.headers.Content-Type: 'text/html' transaction.response.headers.Date: 'Fri, 28 Jul 2023 12:54:08 GMT' transaction.response.headers.ETag: '"64b7baef-1c4"' transaction.response.headers.Last-Modified: 'Wed, 19 Jul 2023 10:29:03 GMT' transaction.response.headers.Server: 'nginx' transaction.response.http_code: '200' transaction.server_id: '08009b29418b10ad42d8757275cce1b4df950f9a' transaction.time_stamp: 'Fri Jul 28 12:54:08 2023' transaction.unique_id: '169054884881.853196'

If you don’t fix the log field format within your JSON, It won’t work.

​

[Dhiraj Ambigapathi]

So the command 'Docker logs container' gives JSON output. Is there anyway to pipe this to Wazuh?
Also in 1st message I've created a decoder with regex  but it is not working

[Luis González Romero]

Sorry, but I think I’m missing something here. Let’s summarize your issue.

You shared a JSON log that contains a log field with escaped characters, making it an invalid JSON format. As a result, it cannot be decoded as a JSON object, and it is treated as a regular string. If the log is generated using the Docker logs container command, you should review its output to verify if that’s a valid JSON format (for example, using https://jsonlint.com/), without escaped characters.

This is the log field content, that has the special chars escaped, e.g \"transaction\"

{\"transaction\":{\"client_ip\":\"187.145.96.30\",\"time_stamp\":\"Fri Jul 28 12:54:08 2023\",\"server_id\":\"08009b29418b10ad42d8757275cce1b4df950f9a\",\"client_port\":32791,\"host_ip\":\"192.168.0.6\",\"host_port\":80,\"unique_id\":\"169054884881.853196\",\"request\":{\"method\":\"GET\",\"http_version\":1.1,\"uri\":\"/\",\"headers\":{\"Host\":\"18.235.43.100:80\",\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36\",\"Content-Length\":\"0\"}},\"response\":{\"body\":\"\u003c!DOCTYPE html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\\n\u003chead\u003e\\n \u003cmeta charset=\\\"UTF-8\\\" /\u003e\\n \u003clink rel=\\\"icon\\\" href=\\\"/favicon.svg\\\" type=\\\"image/svg+xml\\\" /\u003e\\n \u003cmeta name=\\\"viewport\\\" content=\\\"width=device-width, initial-scale=1.0\\\" /\u003e\\n \u003cscript src=\\\"./ckeditor/ckeditor.js\\\"\u003e\u003c/script\u003e\\n \u003cscript type=\\\"module\\\" crossorigin src=\\\"/assets/index.18298ef0.js\\\"\u003e\u003c/script\u003e\\n \u003clink rel=\\\"stylesheet\\\" href=\\\"/assets/index.76e4662b.css\\\"\u003e\\n\u003c/head\u003e\\n\\n\u003cbody\u003e\\n \u003cdiv id=\\\"app\\\"\u003e\u003c/div\u003e\\n \\n\u003c/body\u003e\\n\\n\u003c/html\u003e\\n\",\"http_code\":200,\"headers\":{\"Server\":\"nginx\",\"Date\":\"Fri, 28 Jul 2023 12:54:08 GMT\",\"Content-Length\":\"452\",\"Content-Type\":\"text/html\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Connection\":\"keep-alive\",\"ETag\":\"\\\"64b7baef-1c4\\\"\",\"Accept-Ranges\":\"bytes\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.8 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"Enabled\",\"components\":[\"OWASP_CRS/3.3.4\\\"\"]},\"messages\":[{\"message\":\"Host header is a numeric IP address\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `^[\\\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `18.235.43.100:80' )\",\"reference\":\"o0,16v21,16\",\"ruleId\":\"920350\",\"file\":\"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\",\"lineNumber\":\"719\",\"data\":\"18.235.43.100:80\",\"severity\":\"4\",\"ver\":\"OWASP_CRS/3.3.4\",\"rev\":\"\",\"tags\":[\"modsecurity\",\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-protocol\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/210/272\",\"PCI/6.5.10\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}\n

If you have no control over the log format and it cannot be changed, you will need to create a script or implement a solution to transform the log into a valid JSON format before processing it with Wazuh.

Please let me know if I misunderstood anything or if you need further clarification.

​

[Dhiraj Ambigapathi]

Hi Luiz,
I had three issues
Issue 1
I know we cannot parse log string (the escaped json log) since I tested with wazuh-logtest. But when I run Docker logs container, it provides me a JSON format. As mentioned below
{\"transaction\":{\"client_ip\":\"187.145.96.30\",\"time_stamp\":\"Fri Jul 28 12:54:08 2023\",\"server_id\":\"08009b29418b10ad42d8757275cce1b4df950f9a\",\"client_port\":32791,\"host_ip\":\"192.168.0.6\",\"host_port\":80,\"unique_id\":\"169054884881.853196\",\"request\":{\"method\":\"GET\",\"http_version\":1.1,\"uri\":\"/\",\"headers\":{\"Host\":\"18.235.43.100:80\",\"User-Agent\":\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36\",\"Content-Length\":\"0\"}},\"response\":{\"body\":\"\u003c!DOCTYPE html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\\n\u003chead\u003e\\n \u003cmeta charset=\\\"UTF-8\\\" /\u003e\\n \u003clink rel=\\\"icon\\\" href=\\\"/favicon.svg\\\" type=\\\"image/svg+xml\\\" /\u003e\\n \u003cmeta name=\\\"viewport\\\" content=\\\"width=device-width, initial-scale=1.0\\\" /\u003e\\n \u003cscript src=\\\"./ckeditor/ckeditor.js\\\"\u003e\u003c/script\u003e\\n \u003cscript type=\\\"module\\\" crossorigin src=\\\"/assets/index.18298ef0.js\\\"\u003e\u003c/script\u003e\\n \u003clink rel=\\\"stylesheet\\\" href=\\\"/assets/index.76e4662b.css\\\"\u003e\\n\u003c/head\u003e\\n\\n\u003cbody\u003e\\n \u003cdiv id=\\\"app\\\"\u003e\u003c/div\u003e\\n \\n\u003c/body\u003e\\n\\n\u003c/html\u003e\\n\",\"http_code\":200,\"headers\":{\"Server\":\"nginx\",\"Date\":\"Fri, 28 Jul 2023 12:54:08 GMT\",\"Content-Length\":\"452\",\"Content-Type\":\"text/html\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Last-Modified\":\"Wed, 19 Jul 2023 10:29:03 GMT\",\"Connection\":\"keep-alive\",\"ETag\":\"\\\"64b7baef-1c4\\\"\",\"Accept-Ranges\":\"bytes\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.8 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"Enabled\",\"components\":[\"OWASP_CRS/3.3.4\\\"\"]},\"messages\":[{\"message\":\"Host header is a numeric IP address\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `^[\\\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `18.235.43.100:80' )\",\"reference\":\"o0,16v21,16\",\"ruleId\":\"920350\",\"file\":\"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\",\"lineNumber\":\"719\",\"data\":\"18.235.43.100:80\",\"severity\":\"4\",\"ver\":\"OWASP_CRS/3.3.4\",\"rev\":\"\",\"tags\":[\"modsecurity\",\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-protocol\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/210/272\",\"PCI/6.5.10\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}\n So my doubt was cant we periodically run command 'Docker logs Container' and send its output to Wazuh Agent who'll send logs to Wazuh Manager.
Issue 2
I've a python script which converts the JSON escaped log file into proper JSON file, can I run the python script periodically via Wazuh Agent, kind of like a pre-decoder.
Issue 3
I've already created a Decoder for the JSON escaped log format and tested it via Regex 101 and it worked, but my Decoder is not working.

Decoder -

<decoder name="ax">
    <prematch type="pcre2">^\{"log":</prematch>
    <regex offset="after_prematch" type="pcre2">{"log":"(\d+.\d+.\d+.\d+).-.-.\[(\d+\/\w+\/\w+):(\d+:\d+:\d+)\s.....\]\s."(GET|POST)\s(\s+|\S+)\s(HTTP\/\d+..)\\"\s(\d+)\s(\d+)\s........([a-zA-Z0-9. \/\(;_\),]+)\s([a-zA-Z0-9. \/\\]+)[\s+|\S+]+</regex>
    <order>attackerip,date,time,HTTP_Method,URI,HTTP_Protocol,HTTP_Status,Response_Length,Browser</order>
</decoder>

log -

{"log":"135.125.246.189 - - [31/Jul/2023:09:29:08 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.
0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-31T09:29:08.855788269Z"}

Regex101

Apologies to combine too many queries in one.
Let me know if you understood the issues.

[Luis González Romero]

I know we cannot parse log string (the escaped json log) since I tested with wazuh-logtest. But when I run Docker logs container, it provides me a JSON format. As mentioned below

I don’t understand this. If you mean that your log comes as a JSON, you would not need a custom decoder because the native JSON decoder would decode the log. So I am so lost here.

Then, as I think your log is not a JSON, since you only shared a log with a field that is not a JSON (As I said before, you can’t create a custom decoder becaue it is decoded by the native JSON decoder) I would recommend this workaround (in addition to the ones I explained in previous messages):

1. Create a syslog format log using localfile. To do this, you have to add this block into the ossec.conf

   <localfile> <log_format>syslog</log_format> <location>/var/log/custom/midjson.log</location> <out_format>$(timestamp) $(hostname) MidJSON: $(log)</out_format> </localfile>

   You should adapt this to your case, I used the /var/log/custom/midjson.log as the log file where the log is written. They key here is the out_format, what will change the log format so the native JSON decoder does not match now.

2. After restarting, you’ll be able to obtain the events like this now:

   Aug 3 09:57:15 manager44 MidJSON: {"log":"135.125.246.189 - - [31/Jul/2023:09:29:08 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-31T09:29:08.855788269Z"}

3. Add these decoders. One that allows to match using the program_name and then create sibling decoders to decode the pieces you require

   <decoder name="ax"> <program_name>MidJSON</program_name> </decoder> <decoder name="ax"> <parent>ax</parent> <prematch>^\p"log":"</prematch> <regex offset="after_prematch">(\d+.\d+.\d+.\d+)</regex> <order>ip</order> </decoder>

Then, you are ready to go and create your own sibling decoders as you need.

Here you have some references that may help you:

• https://documentation.wazuh.com/current/user-manual/ruleset/custom.html
• https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/sibling-decoders.html
• https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/decoders.html
• https://documentation.wazuh.com/current/user-manual/ruleset/testing.html

Hope this helps you,
Luis

​

[Dhiraj Ambigapathi]

Hi Luis,
I created the decoder as mentioned

<decoder name="ax">
     <prematch type="pcre2">^\{"log":</prematch>

</decoder>

<decoder name="ax1">
    <parent>ax</parent>
    <regex type="pcre2">^\{"log":"(\d+.\d+.\d+.\d+).-.-.\[(\d+\/\w+\/\w+):(\d+:\d+:\d+)\s.....\]\s."(GET|POST)\s(\s+|\S+)\s(HTTP\/\d+..)\\"\s(\d+)\s(\d+)\s........([a-zA-Z0-9. \/\(;_\),]+)\s([a-zA-Z0-9. \/\\]+)[\s+|\S+]+</regex>
    <order>attackerip,date,time, HTTP-Method, URI, HTTP-Protocol, HTTP-Status, Response-Length, Browser, Value</order>
</decoder>

Log:
{"log":"83.97.73.87 - - [03/Aug/2023:08:16:12 +0000] \"GET / HTTP/1.1\" 200 452 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-08-03T08:16:12.860567171Z"}

It works on regex101

But it doesn't work when I try it on wazuh-logtest

I realize JSON parser is triggered when I run on Wazuh-logtest, but my decoder doesn't trigger, I don't know why.

[Luis González Romero]

You have not made the changes to the log using `out_format` as the first two steps of the suggested workaround. Since the log you are using is the one before without these changes, It is being matched by the native JSON decoder.

The purpose of the workaround is to include additional fields, such as the `program_name`, so that you can create your custom decoder using this information.

[Dhiraj Ambigapathi]

Hi Luis,
I will try changing ossec.conf at wazuh agent and then detecting.
Can we change decoder priority order in wazuh?

[Dhiraj Ambigapathi]

Hi Luis,
I added the lines to wazuh agent ossec.conf, but I don't know the output logs to write a decoder. Can we get the output logs after the we add timestamp and hostname?

Ossec.conf

wazuh ossec.log

[Dhiraj Ambigapathi]

Hi Luis,

I went through documentation and came up with the following configuration.
Since the command 'Docker logs container' outputs a JSON file can we do this ?
I'm not getting any logs, so was wondering how this would work.

[Dhiraj Ambigapathi]

Hi Luis,
Any update on this

[Luis González Romero]

Hello again Dhiraj.

I added the lines to wazuh agent ossec.conf, but I don’t know the output logs to write a decoder. Can we get the output logs after the we add timestamp and hostname?

You should enable the logall option in your ossec.conf to be able to see them in the /var/ossec/logs/archives/archives.json|log

​

[Dhiraj Ambigapathi]

Hi Luis,
I tested logall option, got logs. Tried to test with logtest with following decoder

 <decoder name="ax">
     <program_name>MidJSON</program_name>
 </decoder>

 <decoder name="ax">
     <parent>ax</parent>

     <prematch type= "pcre2">\{"log":"</prematch>

     <regex offset="after_prematch">(\d+.\d+.\d+.\d+)</regex>
     <order>ip</order>
 </decoder>

It doesn't decode the log

Sample Log
2023 Aug 16 11:32:36 (Server) any->/var/lib/docker/containers/f0ac89/e05f0ac89-json.log Aug 16 11:32:36 ip-172-31-16-42 MidJSON: {"log":"192.168.0.6 - - [16/Aug/2023:11:32:35 +000
0] \"GET /healthz HTTP/1.0\" 200 452 \"-\" \"healthcheck\"\n","stream":"stdout","time":"2023-08-16T11:32:35.357813808Z"}

[Luis González Romero]

Hello again Dhiraj, I hope you’re doing well.

I tried your decoders and they appear to be working fine.

Aug 3 09:57:15 manager44 MidJSON: {"log":"135.125.246.189 - - [31/Jul/2023:09:29:08 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-31T09:29:08.855788269Z"} **Phase 1: Completed pre-decoding. full event: 'Aug 3 09:57:15 manager44 MidJSON: {"log":"135.125.246.189 - - [31/Jul/2023:09:29:08 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-31T09:29:08.855788269Z"}' timestamp: 'Aug 3 09:57:15' hostname: 'manager44' program_name: 'MidJSON' **Phase 2: Completed decoding. name: 'ax' parent: 'ax' ip: '135.125.246.189'

It appears that you copied the event with an initial space at the start.

If you need further assistance or have any doubts, please don’t hesitate to ask.

Have a great day,
Luis.

​

[Dhiraj Ambigapathi]

Hi Luis,

The Decoders and Rule are working in logtest, but nothing is published on web console

Web Console

[Luis González Romero]

Hello again.

Did you restart after the changes in the ruleset? The logtest tool does not need the restart to apply the changes but the alerts do

Did you verify that new events arrive (tcp/udp/file)? also, do you see something within the /var/ossec/logs/archives/archives.log|json or /var/ossec/logs/alerts/alerts.log|json?

​

[Dhiraj Ambigapathi]

Hi Luis,

I tried a different method and it worked. I don't know why out_format was not working.
decoder
<decoder name="web-sibling">
  <parent>json</parent>
  <use_own_name>true</use_own_name>
  <regex>client_ip\\":\\"(\S+)\\"</regex>
  <order>client_ip</order>
 </decoder>

<decoder name="web-sibling">
  <parent>json</parent>
  <use_own_name>true</use_own_name>
  <regex>server_id\\":\\"(\S+)\\"</regex>
  <order>server_id</order>
 </decoder>

[Alejandro Ruiz Becerra]

Hello Dhiraj

I'll be following your case from now on.

Is your issues resolved? Do you need further assistance?

Regards
Alex

[Dhiraj Ambigapathi]

Hi Alejandro,

The issue is resolved now

[Alejandro Ruiz Becerra]

Good news!

Have a nice week

Regards,

Alex