Akamai Logs decoder

[26ayush...@gmail.com]

Hello Team,

Could you please help me to create a decoder and rules for the Akamai logs? These are in json format.

I've tried ossec-logtest but it doesn't decode it properly.

Please find the attached sample logs.

Thanks!

Best Regards,

Ayush Agarwal

[victor....@wazuh.com]

Hello 26ayush,

JSON decoder should work well for this case. However, your event is too large. The standard input in the case of a console is limited by N_TTY_BUF_SIZE (65538 bytes). This is the reason your event is not decoded correctly using the logtest tool.

To troubleshoot this issue you should store your event in a file and use it as input:

/var/ossec/bin/wazuh-logtest < event

At this point, you can create a basic alert like the following and construct your Akamai ruleset using this as a parent.

  <rule id="101001" level="0">
   <decoded_as>json</decoded_as>
   <field name="type">akamai_siem</field>
   <description>Akamai SIEM event</description>
  </rule>

Using wazuh-logtest

. . . 
**Phase 3: Completed filtering (rules).
        id: '101001'
        level: '0'
        description: 'Akamai SIEM event'

If you have any doubt do not hesitate to ask.

​

[26ayush...@gmail.com]

Hello Victor,

Thank you for your help. I was able to decode these logs and it has started to flow in Kibana as well. However, as some parts of these logs are base_64 encoded, these must first be decoded by SIEM/Wazuh. 

Please see an example at https://docstraining.akamaized.net/tools/siem_decoder.html

The logs get completed decoded here but in Wazuh these remain in base_64 and the same in Kibana. Is there a way to get the complete encoded details in wazuh as well?

Sample Event:

{"type":"akamai_siem","format":"json","version":"1.0","attackData":{"configId":"55210","policyId":"qik1_98229","clientIP":"122.161.48.222","rules":"MzAwMDE4MA%3d%3d%3bUE9MSUNZLUFOT01BTFk%3d%3b","ruleVersions":"MQ%3d%3d%3bMQ%3d%3d%3b","ruleMessages":"UGFydGlhbCBSZXF1ZXN0IEJvZHkgSW5zcGVjdGlvbiBXYXJuaW5nIC0gUmVxdWVzdCBCb2R5IGlzIGxhcmdlciB0aGFuIHRoZSBjb25maWd1cmVkIGluc3BlY3Rpb24gbGltaXQ%3d%3bQW5vbWFseSBTY29yZSBFeGNlZWRlZCBmb3IgV2ViIFBPTElDWSBWaW9sYXRpb24%3d%3b","ruleTags":"QVNFL1dFQl9BVFRBQ0svUE9MSUNZ%3bQVNFL1dFQl9BVFRBQ0svUE9MSUNZ%3b","ruleData":"MTUxMDg%3d%3bVmVjdG9yIFNjb3JlOiAxMDAwLCBHcm91cCBUaHJlc2hvbGQ6IDksIFRyaWdnZXJlZCBSdWxlczogMzAwMDE4MCwgVHJpZ2dlcmVkIFNjb3JlczogMTAwMCwgVHJpZ2dlcmVkIFNlbGVjdG9yOiBSRVFVRVNUX0hFQURFUlM6Q29udGVudC1MZW5ndGgsIE1pdGlnYXRlZCBSdWxlczogLCBMYXN0IE1hdGNoZWQgTWVzc2FnZTog%3b","ruleSelectors":"UkVRVUVTVF9IRUFERVJTOkNvbnRlbnQtTGVuZ3Ro%3bUkVRVUVTVF9IRUFERVJTOkNvbnRlbnQtTGVuZ3Ro%3b","ruleActions":"YWxlcnQ%3d%3bYWxlcnQ%3d%3b"},"httpMessage":{"requestId":"8345f2","start":"1648703117","protocol":"h2","tls":"tls1.3","method":"POST","host":"abbv.sirioncloud.com","port":"443","path":"/contracttemplate/edit","query":"version=2.0","requestHeaders":"Host%3a%20abbv.sirioncloud.com%0d%0aContent-Length%3a%2015108%0d%0aSec-CH-UA%3a%20%22%20Not%20A%3bBrand%22%3bv%3d%2299%22,%20%22Chromium%22%3bv%3d%2299%22,%20%22Google%20Chrome%22%3bv%3d%2299%22%0d%0ax-csrf-token%3a%20null%0d%0asec-ch-ua-mobile%3a%20%3f0%0d%0aUser-Agent%3a%20Mozilla%2f5.0%20(Windows%20NT%2010.0%3b%20Win64%3b%20x64)%20AppleWebKit%2f537.36%20(KHTML,%20like%20Gecko)%20Chrome%2f99.0.4844.84%20Safari%2f537.36%0d%0anewrelic%3a%20eyJ2IjpbMCwxXSwiZCI6eyJ0eSI6IkJyb3dzZXIiLCJhYyI6IjIzMTcwNDMiLCJhcCI6IjI3OTgwMzI4OSIsImlkIjoiZWRkMDQ3MTRkZTA2MGZiOCIsInRyIjoiMDk4OGVlNTI2ZDU1MWM1YyIsInRpIjoxNjQ4NzAzMTQxNzY5fX0%3d%0d%0aContent-Type%3a%20application%2fjson%3bcharset%3dUTF-8%0d%0aAccept%3a%20application%2fjson,%20text%2fjavascript,%20*%2f*%3b%20q%3d0.01%0d%0aCache-Control%3a%20no-cache%0d%0ax-requested-with%3a%20XMLHttpRequest%0d%0asec-ch-ua-platform%3a%20%22Windows%22%0d%0aOrigin%3a%20https%3a%2f%2fabbv.sirioncloud.com%0d%0asec-fetch-site%3a%20same-origin%0d%0asec-fetch-mode%3a%20cors%0d%0asec-fetch-dest%3a%20empty%0d%0aReferer%3a%20https%3a%2f%2fabbv.sirioncloud.com%2fux%2f%0d%0aAccept-Encoding%3a%20gzip,%20deflate,%20br%0d%0aAccept-Language%3a%20en-US,en%3bq%3d0.9%0d%0aCookie%3a%20wfx_unq%3dKllyfCp8yTjBWr5c%3b%20isClientAdmin%3dfalse%3b%20ssoLogin%3dfalse%3b%20abVersion%3db3JpZ2luYWw%3d%3b%20refreshToken%3deyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwYXlsb2FkIjoiYWpnWU83emRGYXM1aW0rTGZRc2w4Ull1V1k1a3M3TXh0bk1QUUwyZHhWY0lOQTVqNzQrS3hCeWtkcDJjK1YreUtGS05oRTJKbHRZNWQ2YWE2OEJaNFArcjYwc3J6YUp3dHF0UDlyakEyeWs1djhxVWhkYncyVjBwV2l6dWwxdFRxeWYyL0paUk1wWXNoK1YrcVlYc2UyeFRlNjFNNWt1SGNyZzBxVjF0YnVNbU8vWnBTYi8yQmVIRXJtRitiTkE5TlRSRzJtU3RjcDA2dnRrWVBiQT0iLCJpc3MiOiJzaXJpb24iLCJleHAiOjE2NDg3MjA1ODV9.3LRmlzq-wnDwLQYFd3eiUu_ImrWBiCPlD1oQb7YfNH4%3b%20_hp2_ses_props.2138726372%3d%257B%2522r%2522%253A%2522https%253A%252F%252Fabbv.sirioncloud.com%252Fux%252Flogin%252F%253FsavedRequest%253D%252F%252523%252Fdashboard%2522%252C%2522ts%2522%253A1648702598560%252C%2522d%2522%253A%2522abbv.sirioncloud.com%2522%252C%2522h%2522%253A%2522%252Fux%252F%2522%252C%2522g%2522%253A%2522%2523%252FlistRender%252F273%2522%257D%3b%20accessToken%3deyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwYXlsb2FkIjoiYWpnWU83emRGYXM1aW0rTGZRc2w4Ull1V1k1a3M3TXh0bk1QUUwyZHhWY0lOQTVqNzQrS3hCeWtkcDJjK1YreUtGS05oRTJKbHRZNWQ2YWE2OEJaNFArcjYwc3J6YUp3dHF0UDlyakEyeWs1djhxVWhkYncyVjBwV25PMzJVSU03Q0RSNEpSZ0FvSnI5TEJ3c1lMMFoyVU5LcXRQOTFxY2Jya0txVkJvYnV3U2VMVnJVN0wrV3BhTU91dnl3MXBuQkNwbmVlNENvdTQ9IiwiaXNzIjoic2lyaW9uIiwiZXhwIjoxNjQ4NzAzNDk3LCJsb2dpbk1lY2hhbmlzbSI6MX0.hHusnb_VipMnylzg_o71VeHOoSPfGWLiaZ4mT8_g0c4%3b%20Authorization%3deyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJwYXlsb2FkIjoiYWpnWU83emRGYXM1aW0rTGZRc2w4Ull1V1k1a3M3TXh0bk1QUUwyZHhWY0lOQTVqNzQrS3hCeWtkcDJjK1YreUtGS05oRTJKbHRZNWQ2YWE2OEJaNFArcjYwc3J6YUp3dHF0UDlyakEyeWs1djhxVWhkYncyVjBwV25PMzJVSU03Q0RSNEpSZ0FvSnI5TEJ3c1lMMFoyVU5LcXRQOTFxY2Jya0txVkJvYnV3U2VMVnJVN0wrV3BhTU91dnl3MXBuQkNwbmVlNENvdTQ9IiwiaXNzIjoic2lyaW9uIiwiZXhwIjoxNjQ4NzAzNDk3LCJsb2dpbk1lY2hhbmlzbSI6MX0.hHusnb_VipMnylzg_o71VeHOoSPfGWLiaZ4mT8_g0c4%3b%20RT%3d%22z%3d1%26dm%3dsirioncloud.com%26si%3dk53pw73gtbo%26ss%3dl1dqq84z%26sl%3d1%26tt%3d0%26obo%3d1%26ld%3dsjppy%26r%3de0caca7839181bd7abd628669f432a4a%26hd%3dsjpq0%22%3b%20_hp2_id.2138726372%3d%257B%2522userId%2522%253A%25221565369316019334%2522%252C%2522pageviewId%2522%253A%25222312839719638266%2522%252C%2522sessionId%2522%253A%25222187409007083546%2522%252C%2522identity%2522%253A%25221012_58113%2522%252C%2522trackerVersion%2522%253A%25224.0%2522%252C%2522identityField%2522%253Anull%252C%2522isIdentified%2522%253A1%257D%3b%20AWSALB%3dw69VrhBov+ahWai+TI3WMbmjZhrbwa0IC%2fdIW5qjYgw%2fpF+%2fcHo%2f0Kx%2fcwM03h5JSPtObjR0VdUhBKwDL0tkfGY4yr4Gj8qitUaoHsEBrYme8lRz6phwlJWWDgV3syICzFL3RndiNts7gDMJcN1QCZeAo7uLPeJYX5T8BDAuCcQKTB7DWQNk3RgqkbXkWA%3d%3d%3b%20AWSALBCORS%3dw69VrhBov+ahWai+TI3WMbmjZhrbwa0IC%2fdIW5qjYgw%2fpF+%2fcHo%2f0Kx%2fcwM03h5JSPtObjR0VdUhBKwDL0tkfGY4yr4Gj8qitUaoHsEBrYme8lRz6phwlJWWDgV3syICzFL3RndiNts7gDMJcN1QCZeAo7uLPeJYX5T8BDAuCcQKTB7DWQNk3RgqkbXkWA%3d%3d%0d%0a","status":"200","bytes":"183","responseHeaders":"Content-Type%3a%20application%2fjson%3bcharset%3dUTF-8%0d%0aContent-Length%3a%20183%0d%0aAccess-Control-Allow-Origin%3a%20https%3a%2f%2fmobile.sirioncdn.com%0d%0aAccess-Control-Allow-Methods%3a%20GET,%20POST,%20PUT,%20HEAD,%20OPTIONS%0d%0aAccess-Control-Allow-Headers%3a%20Origin,%20Accept,%20X-Requested-With,%20Content-Type,%20Access-Control-Request-Method,%20Access-Control-Request-Headers,%20Authorization,%20fullstream,%20bytes,%20sirionmobile,%20refreshToken%0d%0aAccess-Control-Expose-Headers%3a%20Access-Control-Allow-Origin,%20Access-Control-Allow-Credentials%0d%0aAccess-Control-Allow-Credentials%3a%20true%0d%0aAccess-Control-Max-Age%3a%201000%0d%0aContent-Security-Policy%3a%20frame-src%20%27self%27%20https%3a%2f%2f*.force.com%20null%0d%0aX-CONTENT-SECURITY-POLICY%3a%20frame-src%20%27self%27%20https%3a%2f%2f*.force.com%20null%0d%0aX-XSS-Protection%3a%201%3b%20mode%3dblock%0d%0aX-Content-Type-Options%3a%20nosniff%0d%0aX-UA-Compatible%3a%20IE%3dEdge%0d%0aServer%3a%20%0d%0aExpires%3a%20Thu,%2031%20Mar%202022%2005%3a05%3a17%20GMT%0d%0aCache-Control%3a%20max-age%3d0,%20no-cache,%20private%0d%0aDate%3a%20Thu,%2031%20Mar%202022%2005%3a05%3a17%20GMT%0d%0aConnection%3a%20keep-alive%0d%0aSet-Cookie%3a%20AWSALB%3dv4cbwPXBn%2fFqT4dGjC7tnx0SX8DwYa4mKL0ZFnfWGOFhSR6CAyoOLAOxo9oFnG3nLTdJ1PnRix1ob1hg2hynFvJLs8TLY%2f1AEXOJIuMXW9goXQ772niIYa84ZCFgwZnaEs8955KCixx3W6nK3gp7uaengAJK0y0GZlCfuGKnbbuaTU9MBKkW2RSnFVhdZg%3d%3d%3b%20Expires%3dThu,%2007%20Apr%202022%2005%3a05%3a16%20GMT%3b%20Path%3d%2f%0d%0aSet-Cookie%3a%20AWSALBCORS%3dv4cbwPXBn%2fFqT4dGjC7tnx0SX8DwYa4mKL0ZFnfWGOFhSR6CAyoOLAOxo9oFnG3nLTdJ1PnRix1ob1hg2hynFvJLs8TLY%2f1AEXOJIuMXW9goXQ772niIYa84ZCFgwZnaEs8955KCixx3W6nK3gp7uaengAJK0y0GZlCfuGKnbbuaTU9MBKkW2RSnFVhdZg%3d%3d%3b%20Expires%3dThu,%2007%20Apr%202022%2005%3a05%3a16%20GMT%3b%20Path%3d%2f%3b%20SameSite%3dNone%3b%20Secure%0d%0aServer-Timing%3a%20cdn-cache%3b%20desc%3dMISS%0d%0aServer-Timing%3a%20edge%3b%20dur%3d254%0d%0aServer-Timing%3a%20origin%3b%20dur%3d1068%0d%0aStrict-Transport-Security%3a%20max-age%3d15768000%20%3b%20includeSubDomains%20%3b%20preload%0d%0a"},"geo":{"continent":"AS","country":"IN","city":"NEWDELHI","regionCode":"DL","asn":"24560"}}

Thanks!

Best Regards,

Ayush Agarwal

[victor....@wazuh.com]

Hello Ayush, currently there is no native solution for this issue. There is an open issue to include this feature https://github.com/wazuh/wazuh/issues/6817.

The best approach is to use an external tool to decode those logs into a monitoring file.

Let me know if you have any doubts about how to implement this workflow and I will help you with that.

[26ayush...@gmail.com]

Hello Victor,

Surely I will need your help with this. Please let me know what needs to be done for this such as any open-source external decoder and the workflow.

Best,

Ayush Agarwal

[victor....@wazuh.com]

Hello Ayush, and sorry for the late reply.

In order to parse those logs we need to follow these steps:

• Configure rsyslog to store your undecoded Akamai logs into a file, for example, umparsed_file.json.

• Create a custom script, or use an external tool, to decode those logs every time a new event is recorded.
  The following script will decode the json data  attackData of your events based on the example you shared. For the rest of the fields, you will need to edit the script.

#!/usr/bin/python
import os
import json
import base64
import re
import time
import argparse

encoded_fields = ['rules', 'ruleVersions', 'ruleMessages', 'ruleTags', 'ruleData', 'ruleSelectors', 'ruleActions']

def decode_base64(data, altchars=b'+/'):
    data = re.sub(rb'[^a-zA-Z0-9%s]+' % altchars, b'', data)
    missing_padding = len(data) % 4
    if missing_padding:
        data += b'='* (4 - missing_padding)
    return base64.b64decode(data, altchars)

def decode_attackData(dct):
        decoded_dict = dct
        if "attackData" in dct:
                for key,value in dct['attackData'].items():
                        if key in encoded_fields:
                                encoded_list = [value]
                                encoded_value = value
                                decoded_list = []
                                if '%3d%3b' in encoded_value:
                                        encoded_list = encoded_value.split('%3d%3b')
                                if '%3d%3d%3b' in encoded_value:
                                        encoded_list = encoded_value.split('%3d%3d%3b')
                                for i in encoded_list:
                                    element = i.replace('%3b', '')
                                    element = element.replace('%3d', '')

                                    decoded_valued = decode_base64(element.encode())
                                    decoded_list.append(decoded_valued.decode('utf-8'))
                                    decoded_dict['attackData'][key] = decoded_list
        return decoded_dict

def tail_file(filename):
    with open(filename) as file:
        file.seek(0, os.SEEK_END)
        while True:
            line = file.readline()
            if not line:
                time.sleep(1)
                continue

            yield line

def main():
    arg_parser = argparse.ArgumentParser()
    arg_parser.add_argument('-i', '--input-file', metavar='<input-file>', type=str, required=True, help='Input undecoded file', dest='input_file')
    arg_parser.add_argument('-o', '--output-file', metavar='<output-file>', type=str, required=True, help='Output file', dest='output_file')
    args = arg_parser.parse_args()

    logs_lines = tail_file(args.input_file)
    for line in logs_lines:
            try:
                json_log = json.loads(line, object_hook=decode_attackData) 
                with open(args.output_file, 'a') as o:
                    o.write(json.dumps(json_log))
                    o.write('\n')

            except Exception:
                print(f"ERROR parsing {line}")

if __name__ == "__main__":
    main()

You can use it as follows:

python3 parser_decode64.py —i <input_file> -o  <output_file>

Where

input_file: Path where akamai undecoded events are stored
output_file: Path where akamai decoded events will be placed. This is the path you need to monitor with the Wazuh agent/manager

Take in mind that this script is a proof of concept, and is probably unstable and not suited for the production environment. Please edit it according to your needs.

If we use logtest as specified in my last message we get all the decoded fields:

**Phase 2: Completed decoding.
    name: 'json'
    attackData.clientIP: '122.161.48.222'
    attackData.configId: '55210'
    attackData.policyId: 'qik1_98229'
    attackData.ruleActions: 'alert,alert,'
    attackData.ruleData: '15108,Vector Score: 1000, Group Threshold: 9, Triggered Rules: 3000180, Triggered Scores: 1000, Triggered Selector: REQUEST_HEADERS:Content-Length, Mitigated Rules: , Last Matched Message: '
    attackData.ruleMessages: 'Partial Request Body Inspection Warning - Request Body is larger than the configured inspection limit,Anomaly Score Exceeded for Web POLICY Violation,'
    attackData.ruleSelectors: 'REQUEST_HEADERS:Content-LengthREQUEST_HEADERS:Content-Length'
    attackData.ruleTags: 'ASE/WEB_ATTACK/POLICYASE/WEB_ATTACK/POLICY'
    attackData.ruleVersions: '1,1,'
    attackData.rules: '3000180,POLICY-ANOMALY'

​

[26ayush...@gmail.com]

Hi Victor,

Thank you for the script. However, during its execution, nothing happens. I've tested it with one sample event, please see the attached screenshot, it does not create an output file and does not return any error as such.

Request you to please guide me further on this.

Best Regards,

Ayush Agarwal

[victor....@wazuh.com]

Yes, that is the expected behavior of the script.

The script will gather every message that is placed into the input file in real-time and it will decode it into the output file.

To test this script do the following:

- Run the script in the background

python3 script.py -i input.txt -o output.txt &

- Place a testing event log into a file with the name example.txt (for example, the event specified in this message https://groups.google.com/g/wazuh/c/iJTxeRrY2Us/m/prX0CpbKBAAJ)
- Run

cat event.txt >> input.txt

You will see the same message decoded in the output file

Remind that this is only a proof of concept and I recommend improving this script for production use.

​