JSON Decoder question

[Jared Greene]

I would like to pull a string fo data from a nested element of decoded json log. 

Example Log:

{
 "division_id":"Marksman-Test",
 "ingest_time":"2018-08-08T16:00:57.000Z",
 "type":"Foreman",
 "action":
   {
 "objectType":"core.user_auth.login_failed",
 "message":"Sign-In Failed - Verification failed for user: mobb...@marksman-test.com",
 "requestUri":"/api/v1/authn",
 "categories":
     [
   "Sign-in Failure",
   "Suspicious Activity"
 ]
 },
 "actors":[
 {
 "objectType":"Client",
 "displayName":"SAFARI",
 "ipAddress":"17.77.29.26",
 "id":"Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1"}
 ],
 "eventId":"MyKsA1525729898000tevgs9cs9eASZORaDR1c",
 "targets":
   [
    {
     "objectType":"User",
 "id":""
    }
   ],
 "sessionId":"",
 "requestId":"4SupNJyLDKTWvC@iKa66qAAACEM",
 "@timestamp":"2018-05-07T21:00:57.000Z",
 "customer_id":"Marksman-Test",
 "published":"2018-08-08T16:00:57.000Z",
 "@version":"1"
}

The sting that I am interested in is for user Mob Barley from the following line. 

"message":"Sign-In Failed - Verification failed for user: mobb...@Marksman-Test.com",

Is there a way to go deeper than action.message and create a field from within a value? 

The desired outcome would be:

 

"action.message.user" = "mobb...@marksman-test.com"
or
"user": "mobb...@marksman-test.com"

from the log above. 

Thank you - J

[Victor Fernandez]

Hi Jared,

I have tried to extract the log from the JSON input and it seems that plugin decoders are not compatible with multi-regex decoders. The idea was to use two decoders: one for JSON (written natively as a plugin decoder) and another for the field you want (using a standard decoder with a regex in XML).

I have made some modifications to the code and reached what you wanted with these decoders —replacing /var/ossec/ruleset/decoders/0006-json_decoders.xml—:

<decoder name="json">
  <prematch>^{\s*"</prematch>
</decoder>

<decoder name="json_child">
  <parent>json</parent>
  <prematch>^{\s*"</prematch>
  <plugin_decoder>JSON_Decoder</plugin_decoder>
</decoder>

<decoder name="json_child">
  <parent>json</parent>
  <regex>Verification failed for user: (\S+)"</regex>
  <order>action.message.user</order>
</decoder>

Then, with your input, these fields are extracted:

decoder: 'json'
division_id: 'Marksman-Test'
ingest_time: '2018-08-08T16:00:57.000Z'
type: 'Foreman'
action.objectType: 'core.user_auth.login_failed'
action.message: 'Sign-In Failed - Verification failed for user: mobb...@marksman-test.com'
action.requestUri: '/api/v1/authn'
action.categories: 'Sign-in Failure,Suspicious Activity,'
eventId: 'MyKsA1525729898000tevgs9cs9eASZORaDR1c'
sessionId: ''
requestId: '4SupNJyLDKTWvC@iKa66qAAACEM'
@timestamp: '2018-05-07T21:00:57.000Z'
customer_id: 'Marksman-Test'
published: '2018-08-08T16:00:57.000Z'
@version: '1'
action.message.user: 'mobb...@marksman-test.com'

Note that the last field is a substring of "action.message" as you wanted.

I have just made a PR (#602) to the branch 3.3 with this functionality. Hope you find it useful.

Best regards,

Victor M Fernandez-Castro 
IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/875bbf30-83dc-46e1-ba1d-d23dd8e6890c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jared Greene]

Perfect! - Thank you! - Jared

--

Thank you,

Jared R. Greene
VP, COO/CISO
Cloud Management International, LLC
p: 407.414.4003
f:  919.882.0956
www.cloudawarefederal.com
www.linkedin.com/in/jaredrgreene
Please consider the environment before printing this email. The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited.

[Jared Greene]

Looking for some additional guidance on this topic. 

1. Does this mean that I need to upgrade to 3.3 across the board where this JSON support is required or is there a patch?

2. Is there a way to call out json array elements using the wazuh decoder?

{"@timestamp":"2018-06-10T18:51:58.000Z","division_id":"test","action":{"objectType":"core.user_auth.login_success","requestUri":"/api/v1/authy","message":"Sign-in successful","categories":["Sign-in Success"]},"targets":[{"displayName":"mob barley","objectType":"User","id":"0nums6","login":"mba...@marksman-test.com"}],"sessionId":"ID","ingest_time":"2018-06-10T18:53:07Z","eventId":"event-id","type":"okta","published":"2018-06-10T18:51:58.000Z","requestId":"request-@id","@version":"1","customer_id":"marksman","actors":[{"displayName":"mob barley","objectType":"User","id":"letters","login":"mba...@marksman-test.com.com"},{"displayName":"CHROME","objectType":"Client","ipAddress":"10.20.30.40","id":"AppleWebKit/537.26 (KHTML, like Gecko) Chrome/66.0.3359.639 Safari/537.26"}]}

 

   "actors":[{"displayName":"mob barley","objectType":"User","id":"0num6","login":"mba...@marksman-test.com"},{"displayName":"CHROME","objectType":"Client","ipAddress":"10.20.30.40","id":"AppleWebKit/537.26 (KHTML, like Gecko) Chrome/66.0.3359.639 Safari/537.26"}]

Desired fields:

<field name="actors.0.displayName">\.+</field> where field  =  mob barley

<field name="actors.0.objectType">\.+</field> where field  = user

<field name="actors.0.id">\.+</field> where field  =  0num6

<field name="actors.0.login">\.+</field> where field  = mba...@marksman-test.com

<field name="actors.1.displayName">\.+</field> where field  =  CHROME 

<field name="actors.1.objectType">\.+</field> where field  = Client

<field name="actors.1.ipaddress">\.+</field> where field  =   10.20.30.40

<field name="actors.1.id">\.+</field> where field  =  AppleWebKit/537.26 (KHTML, like Gecko) Chrome/66.0.3359.639 Safari/537.26

Is there a proper way to reference the data nested in an array using the json decoders to pull out IP addresses from Okta logs for example?

I did read the docs, but can this be a feature?

Arrays

Lists with zero or more values. These values may be different, but they must belong to some of the above. An array of objects is not supported.

Thanks again! - J

[Victor Fernandez]

Hi Jared,

Sorry for the late response. We are still working on the 3.3 development so we don't have yet an upgrade tool. You can install 3.3 from sources or patch 3.2 if you are using that version.

The commit applying the changes is https://github.com/wazuh/wazuh/commit/3bfb09621df17df6d3ec43e9946c1c1802e3c327. If you want to patch the version 3.2 with this commit, you can proceed this way:

# Get Wazuh source code, branch 3.2 (currently 3.2.3 release-candidate)
git clone https://github.com/wazuh/wazuh.git -b 3.2
cd wazuh

# Patch
git cherry-pick 3bfb09621df17df6d3ec43e9946c1c1802e3c327

# Fix conflict in the changelog file. This is optional but helpful to apply the commit.
git checkout --ours CHANGELOG.md
git add CHANGELOG.md
git cherry-pick --continue

# Compile Wazuh server
make -C src -j4 TARGET=server

# Stop Wazuh manager and install only the analysis daemon
systemctl stop wazuh-manager
cp src/ossec-analysisd /var/ossec/bin
systemctl start wazuh-manager

Regarding your last question, we don't have yet any way to reference array indexes. Data is structured in the analysis engine as key-value pairs. The JSON decoder maps the objects this way:

• Nested objects: the field name is the name of all container objects joint by a dot.
• Number arrays: Values are converted into a string with all numbers separated by a comma.
• String arrays: same as number arrays.
  
• Object arrays: they are not parsed.

We want to get rid of this by transforming the internal representation of fields into pure JSON structure. We are working on a new rule condition expression language like:

size(a.b) > 2 && any x in a.b: (x =~ "sg*" || int(x) == 4)
a.b[a.d + 2] == 4 && all x in a.c: (x.value > 7 || x.comment == null || x.children[0] == 2)

I hope to have this language implemented in the next major version.

Hope it help.

Best regards,

Victor M Fernandez-Castro 
IT Engineer — Wazuh, Inc.

On Thu, May 10, 2018 at 1:25 PM, Jared Greene <jgr...@cloudawarefederal.com> wrote:

Looking for some additional guidance on this topic. 

1. Does this mean that I need to upgrade to 3.3 across the board where this JSON support is required or is there a patch?

2. Is there a way to call out json array elements using the wazuh decoder?

{"@timestamp":"2018-06-10T18:51:58.000Z","division_id":"test","action":{"objectType":"core.user_auth.login_success","requestUri":"/api/v1/authy","message":"Sign-in successful","categories":["Sign-in Success"]},"targets":[{"displayName":"mob barley","objectType":"User","id":"0nums6","login":"mbarley@marksman-test.com"}],"sessionId":"ID","ingest_time":"2018-06-10T18:53:07Z","eventId":"event-id","type":"okta","published":"2018-06-10T18:51:58.000Z","requestId":"request-@id","@version":"1","customer_id":"marksman","actors":[{"displayName":"mob barley","objectType":"User","id":"letters","login":"mbarley@marksman-test.com.com"},{"displayName":"CHROME","objectType":"Client","ipAddress":"10.20.30.40","id":"AppleWebKit/537.26 (KHTML, like Gecko) Chrome/66.0.3359.639 Safari/537.26"}]}

 

   "actors":[{"displayName":"mob barley","objectType":"User","id":"0num6","login":"mbarley@marksman-test.com"},{"displayName":"CHROME","objectType":"Client","ipAddress":"10.20.30.40","id":"AppleWebKit/537.26 (KHTML, like Gecko) Chrome/66.0.3359.639 Safari/537.26"}]

[Victor Fernandez]

Actually we don't have a standard procedure to choose rule ID numbers. The order of the IDs does not matter, but the order in which rules are defined does matter, that's why we use number prefixes in the rule files. We usually assign 4-digit, 5-digits or 6-digits IDs for decoded rules, and leave 3-digit identifiers for internal rules from Wazuh (agent connected or disconnected, FIM alerts, etc).

We recommend creating custom rules in etc/rules (instead of ruleset/rules) and use 6-digit IDs starting by "10", just as you will find in etc/rules/local_rules.xml.

Best regards.

Victor M Fernandez-Castro 
IT Engineer — Wazuh, Inc.

On Wed, May 16, 2018 at 7:47 PM, Jared Greene <jgr...@cloudawarefederal.com> wrote:

We were able to get this working from the 3.3 branch. In order to get the nested values from within all of the Okata events, we added a decoder before the default json decoder as follows:

<decoder name="okta">

  <prematch>"type":"okta"</prematch>

</decoder>

We are decoding the rest of the fields vi regex and it is working well. We will keep an eye out for an enhancement that allows for json decoding. 

Completely unrelated question. We now have all 2214 Cisco Firepower Syslog events, Okta decoders, Juniper-SRX firewalls, and Oracle logs mapped to rules and decoders and that we would like to share. How does Wazuh manage or issue Rule Id number ranges to prevent overlap and duplication of id's.  We would like to start pushing rules and decoders, but want to color in the lines and not create a bunch of work and refactoring for rule id ranges.

Thank you!!

Jared 

[Jared Greene]

We were able to get this working from the 3.3 branch. In order to get the nested values from within all of the Okata events, we added a decoder before the default json decoder as follows:

<decoder name="okta">

  <prematch>"type":"okta"</prematch>

</decoder>

We are decoding the rest of the fields vi regex and it is working well. We will keep an eye out for an enhancement that allows for json decoding. 

Completely unrelated question. We now have all 2214 Cisco Firepower Syslog events, Okta decoders, Juniper-SRX firewalls, and Oracle logs mapped to rules and decoders and that we would like to share. How does Wazuh manage or issue Rule Id number ranges to prevent overlap and duplication of id's.  We would like to start pushing rules and decoders, but want to color in the lines and not create a bunch of work and refactoring for rule id ranges.

Thank you!!

Jared