re: AWS Cloudtrail decoder

[mark.f...@gmail.com]

I'm working on a decoder to pull information out of the AWS Cloudtrail logs.  There are several examples online and I used them as a starting point.  

I've successfully pulled three components out of each Cloudtrail log, but am not having success when I add a fourth.  

I've split the regex up to see if each individual component works and they do, but when I combine them all, I'm not having any success (when I test with ossec-logtest.

It could be that I'm missing something simple, but I'm also wondering if there's a limitation to the regex string length.  I haven't looked at the source and am hoping it is something I'm just missing.

Below are both the regex and the sample Cloudtrail log entry I am trying to parse.

Any help would be greatly appreciated.  

sample log entry (it's all one line)

####################################################################

"eventVersion":"1.02","eventID":"c0b8a753-65f8-4ea8-a924-263d4555a725","eventTime":"2015-02-19T19:46:40Z","requestParameters":"{u'maxItems': u'100'}","eventType":"AwsApiCall","responseElements":"None","awsRegion":"us-east-1","eventName":"ListHealthChecks","userIdentity":"{u'userName': u'mark.feferman', u'principalId': u'AIDAJCZ7AIQAFLV3CQZJA', u'accessKeyId': u'ASIAJX7ZFLPD2SJOPQXQ', u'invokedBy': u'signin.amazonaws.com', u'sessionContext': {u'attributes': {u'creationDate': u'2015-02-19T19:43:21Z', u'mfaAuthenticated': u'false'}}, u'type': u'IAMUser', u'arn': u'arn:aws:iam::875787860505:user/mark.feferman', u'accountId': u'875787860505'}","eventSource":"route53.amazonaws.com","requestID":"0634d15d-b870-11e4-827b-6158e160b350","apiVersion":"2013-04-01","userAgent":"signin.amazonaws.com","sourceIPAddress":"192.168.1.1","recipientAccountId":"875787860505"

####################################################################

decoder

####################################################################

<decoder name="cloudtrail">

 <prematch>^"eventVersion":"\d.\d\d"</prematch>

  <!-- the below DOES work and retrieves the awsRegion, username, and IP address -->

  <!-- <regex>"awsRegion":"(\S+)"\.+"eventName":"(\S+)"\.+"userIdentity":"{u'userName': u'(\S+)'</regex> -->

  <!-- the below line DOES NOT work (when I add the sourceIPAddress -->

  <regex>"awsRegion":"(\S+)"\.+"eventName":"(\S+)"\.+"userIdentity":"{u'userName': u'(\S+)'\.+"sourceIPAddress":"(\d+.\d+.\d+.\d+)"</regex> 

 <order>data,action,srcuser,srcip</order>

</decoder>

####################################################################

Sincerely,

Mark

Mark Feferman, CISM, CISSP, CSSLP

Principal @ Vaunted Group

ma...@vauntedgroup.com

713.478.5150

http://www.vauntedgroup.com

Expertise in the discipline of Software Security Assurance

[Mark Feferman]

One thing to note:

I've tried the various components of the regex, in different combinations, and I am successful at pulling in three of the four, including the sourceIPAddress, but when all four are combined, as in my original post, I fail. :(

[David Lang]

question, are you sure this isn't wrapped in {} making it valid json?

If it is, you could use jq to pull it apart and output the things in the
order/format that you want them (to match a parser)

David Lang

On Thu, 28 May 2015, Mark Feferman wrote:

> Date: Thu, 28 May 2015 12:28:45 -0700 (PDT)
> From: Mark Feferman <mark.f...@gmail.com>
> Reply-To: ossec...@googlegroups.com
> To: ossec...@googlegroups.com
> Subject: [ossec-list] Re: AWS Cloudtrail decoder

[Mark Feferman]

It is wrapped in JSON, but with Cloudtrail event logs, there are several per zip file.  The idea was to use a process already outlined online (http://blog.rootshell.be/2013/11/15/keep-an-eye-on-your-amazon-cloud-with-ossec/) to process the Cloudtrail event logs, using them for alerting in OSSEC.  I already have 99% of this working, so I was hoping this regex issue would be simple enough that I wouldn't have to go a different route.

On Thursday, May 28, 2015 at 12:59:13 PM UTC-5, Mark Feferman wrote: