Been using wazuh since last 2 odd weeks. Started forwarding logs from our firewall, was able to make dashboards up and running quickly. And i thought everthing was fine.
However today i noticed bittorrent traffic on the firewall and tried searching for it using wazuh Discovery, to my surprise there was no logs of that traffic. I made sure that all traffic and threat logs are being forwarded. FIltered traffic based on that specific IP from which torrent was being generated, results were fine. From Palo Alto UI logs i saw traffic was being blocked, so i applied the blocked filter and i came to the conclusion that no block logs from that specific IP was available.
Other blocked traffic from Blacklisted IP rules was being shown just fine.
So i furher investigated that many threat log fields are not even available in the Discover section of the Wazuh UI. Fields like "URL Category List, http header, source_device_catgeory,f_address, source_device_category, source_device_profile, source_device_model, source_device_vendor, threat_content_type,payload_protocol_id" and many other fields are not being shown. I further noticed that log related to a lot of other rules was not being displayed either until i manually ssearched for those particular rules using the discover tab.
Now i want to know what's wrong? I dont think there is any configuration that is required since logs are being decoded by the palo alto decoder properly.
Any guidance will be much appreciated.
Few sample logs from archive.json below
{"timestamp":"2023-08-23T03:48:54.907+0000","rule":{"level":2,"description":"Palo Alto Traffic: Session ended on FFPAFW-1 from 172.16.15.16 to 59.103.92.153. Reason: threat.","id":"64507","firedtimes":244929,"mail":false,"groups":["paloalto"],"gdpr":["IV_35.7.d"],"gpg13":["4.12"],"hipaa":["164.312.b"],"pci_dss":["1.4","10.6.1","11.4"],"tsc":["CC6.1","CC6.7","CC6.8","CC7.2","CC7.3","CC7.4"]},"agent":{"id":"000","name":"FF_SIEM"},"manager":{"name":"FF_SIEM"},"id":"1692762534.2196704200","full_log":"Aug 23 08:46:13 FFPAFW-1.ffho.org 1,2023/08/23 08:46:12,024301000859,TRAFFIC,end,2562,2023/08/23 08:46:12,172.16.15.16,59.103.92.153,1.1.1.1,59.103.92.153,LAN-Internet-Staff,,,web-browsing,vsys1,LAN-ZONE,PTCL-Zone,ethernet1/23,ethernet1/2,Syslog-server,2023/08/23 08:46:12,2351663,1,54338,80,50927,80,0x400010,tcp,allow,70373,1115,69258,63,2023/08/23 08:45:57,0,computer-and-internet-info,,7269008771584308918,0x0,172.16.0.0-172.31.255.255,Pakistan,,15,48,threat,0,0,0,0,,FFPAFW-1,from-policy,,,0,,0,,N/A,0,0,0,0,58cf2039-7fe8-43ea-a93f-2d585dca4b08,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2023-08-23T08:46:13.088+05:00,,,internet-utility,general-internet,browser-based,4,\"used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use\",,web-browsing,no,no,0","predecoder":{"timestamp":"Aug 23 08:46:13","hostname":"FFPAFW-1.ffho.org"},"decoder":{"parent":"paloalto","name":"paloalto"},"data":{"protocol":"tcp","action":"allow","receive_time":"2023/08/23 08:46:12","serial_number":"1234512345","type":"TRAFFIC","content_type":"end","generated_time":"2023/08/23 08:46:12","source_address":"172.16.15.16","destination_address":"59.103.92.153","nat_source_ip":"119.156.11.111","nat_destination_ip":"59.103.92.153","rule_name":"LAN-Internet-Staff","application":"web-browsing","virtual_system":"vsys1","source_zone":"LAN-ZONE","destination_zone":"PTCL-Zone","inbound_interface":"ethernet1/23","outbound_interface":"ethernet1/2","log_action":"Syslog-server","session_id":"2351663","repeat_count":"1","source_port":"54338","destination_port":"80","nat_source_port":"50927","nat_destination_port":"80","flags":"0x400010","bytes":"70373","bytes_sent":"1115","bytes_received":"69258","packets":"63","start_time":"2023/08/23 08:45:57","elapsed_time":"0","category":"computer-and-internet-info","sequence_number":"7269008771584308918","action_flags":"0x0","source_country":"172.16.0.0-172.31.255.255","destination_country":"Pakistan","packets_sent":"15","packets_received":"48","session_end_reason":"threat","device_group_hierarchy_level_1":"0","device_group_hierarchy_level_2":"0","device_group_hierarchy_level_3":"0","device_group_hierarchy_level_4":"0","device_name":"FFPAFW-1","action_source":"from-policy","tunnel_id_imsi":"0","parent_session_id":"0","tunnel_type":"N/A","sctp_association_id":"0","sctp_chunks":"0","sctp_chunks_sent":"0","sctp_chunks_received":"0","rule_uuid":"58cf2039-7fe8-43ea-a93f-2d585dca4b08","http_2_connection":"0","app_flap_count":"0","high_resolution_timestamp":"2023-08-23T08:46:13.088+05:00","application_subcategory":"internet-utility","application_category":"general-internet","application_technology":"browser-based","application_risk":"4","application_characteristic":"\"used-by-malware","application_container":"able-to-transfer-file","application_saas":"has-known-vulnerability","application_sanctioned_state":"tunnel-other-application"},"location":"1.1.1.1"}
{"timestamp":"2023-08-23T03:50:19.425+0000","rule":{"level":6,"description":"Palo Alto Traffic: Session dropped on FFPAFW-1 from 46.174.191.28 to 59.103.181.105. Reason: policy-deny. Action: deny.","id":"64508","mitre":{"id":["T1072","T1190"],"tactic":["Execution","Lateral Movement","Initial Access"],"technique":["Software Deployment Tools","Exploit Public-Facing Application"]},"firedtimes":168,"mail":false,"groups":["paloalto"],"gdpr":["IV_35.7.d"],"gpg13":["4.12"],"hipaa":["164.312.b"],"pci_dss":["1.4","10.6.1","11.4"],"tsc":["CC6.1","CC6.7","CC6.8","CC7.2","CC7.3","CC7.4"]},"agent":{"id":"000","name":"FF_SIEM"},"manager":{"name":"FF_SIEM"},"id":"1692762619.2226452957","full_log":"Aug 23 08:47:37 FFPAFW-1.ffho.org 1,2023/08/23 08:47:36,024301000859,TRAFFIC,drop,2562,2023/08/23 08:47:36,46.174.191.28,59.103.181.105,0.0.0.0,0.0.0.0,Black-listed-IPS-Source-Inbound,,,not-applicable,vsys1,PTCL-Zone,PTCL-Zone,ethernet1/2,,Syslog-server,2023/08/23 08:47:36,0,1,38364,8080,0,0,0x0,tcp,deny,0,0,0,1,2023/08/23 08:47:37,0,any,,7269008771584326457,0x0,Ukraine,Pakistan,,1,0,policy-deny,0,0,0,0,,FFPAFW-1,from-policy,,,0,,0,,N/A,0,0,0,0,d000516f-0ba4-4876-93cd-4fca4c5e4813,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2023-08-23T08:47:37.640+05:00,,,unknown,unknown,unknown,1,,,not-applicable,no,no,0","predecoder":{"timestamp":"Aug 23 08:47:37","hostname":"FFPAFW-1.ffho.org"},"decoder":{"parent":"paloalto","name":"paloalto"},"d{"timestamp":"2023-08-23T03:48:54.907+0000","rule":{"level":2,"description":"Palo Alto Traffic: Session ended on FFPAFW-1 from 172.16.15.16 to 59.103.92.153. Reason: threat.","id":"64507","firedtimes":244929,"mail":false,"groups":["paloalto"],"gdpr":["IV_35.7.d"],"gpg13":["4.12"],"hipaa":["164.312.b"],"pci_dss":["1.4","10.6.1","11.4"],"tsc":["CC6.1","CC6.7","CC6.8","CC7.2","CC7.3","CC7.4"]},"agent":{"id":"000","name":"FF_SIEM"},"manager":{"name":"FF_SIEM"},"id":"1692762534.2196704200","full_log":"Aug 23 08:46:13 FFPAFW-1.ffho.org 1,2023/08/23 08:46:12,024301000859,TRAFFIC,end,2562,2023/08/23 08:46:12,172.16.15.16,59.103.92.153,1.1.1.1,59.103.92.153,LAN-Internet-Staff,,,web-browsing,vsys1,LAN-ZONE,PTCL-Zone,ethernet1/23,ethernet1/2,Syslog-server,2023/08/23 08:46:12,2351663,1,54338,80,50927,80,0x400010,tcp,allow,70373,1115,69258,63,2023/08/23 08:45:57,0,computer-and-internet-info,,7269008771584308918,0x0,172.16.0.0-172.31.255.255,Pakistan,,15,48,threat,0,0,0,0,,FFPAFW-1,from-policy,,,0,,0,,N/A,0,0,0,0,58cf2039-7fe8-43ea-a93f-2d585dca4b08,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2023-08-23T08:46:13.088+05:00,,,internet-utility,general-internet,browser-based,4,\"used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use\",,web-browsing,no,no,0","predecoder":{"timestamp":"Aug 23 08:46:13","hostname":"FFPAFW-1.ffho.org"},"decoder":{"parent":"paloalto","name":"paloalto"},"data":{"protocol":"tcp","action":"allow","receive_time":"2023/08/23 08:46:12","serial_number":"024301000859","type":"TRAFFIC","content_type":"end","generated_time":"2023/08/23 08:46:12","source_address":"172.16.15.16","destination_address":"59.103.92.153","nat_source_ip":"1.1.1.1","nat_destination_ip":"59.103.92.153","rule_name":"LAN-Internet-Staff","application":"web-browsing","virtual_system":"vsys1","source_zone":"LAN-ZONE","destination_zone":"PTCL-Zone","inbound_interface":"ethernet1/23","outbound_interface":"ethernet1/2","log_action":"Syslog-server","session_id":"2351663","repeat_count":"1","source_port":"54338","destination_port":"80","nat_source_port":"50927","nat_destination_port":"80","flags":"0x400010","bytes":"70373","bytes_sent":"1115","bytes_received":"69258","packets":"63","start_time":"2023/08/23 08:45:57","elapsed_time":"0","category":"computer-and-internet-info","sequence_number":"7269008771584308918","action_flags":"0x0","source_country":"172.16.0.0-172.31.255.255","destination_country":"Pakistan","packets_sent":"15","packets_received":"48","session_end_reason":"threat","device_group_hierarchy_level_1":"0","device_group_hierarchy_level_2":"0","device_group_hierarchy_level_3":"0","device_group_hierarchy_level_4":"0","device_name":"FFPAFW-1","action_source":"from-policy","tunnel_id_imsi":"0","parent_session_id":"0","tunnel_type":"N/A","sctp_association_id":"0","sctp_chunks":"0","sctp_chunks_sent":"0","sctp_chunks_received":"0","rule_uuid":"58cf2039-7fe8-43ea-a93f-2d585dca4b08","http_2_connection":"0","app_flap_count":"0","high_resolution_timestamp":"2023-08-23T08:46:13.088+05:00","application_subcategory":"internet-utility","application_category":"general-internet","application_technology":"browser-based","application_risk":"4","application_characteristic":"\"used-by-malware","application_container":"able-to-transfer-file","application_saas":"has-known-vulnerability","application_sanctioned_state":"tunnel-other-application"},"location":"1.1.1.1"}
{"timestamp":"2023-08-23T03:50:05.603+0000","rule":{"level":2,"description":"Palo Alto Traffic: Session ended on FFPAFW-1 from 172.16.22.46 to 10.1.0.51. Reason: aged-out.","id":"64507","firedtimes":253967,"mail":false,"groups":["paloalto"],"gdpr":["IV_35.7.d"],"gpg13":["4.12"],"hipaa":["164.312.b"],"pci_dss":["1.4","10.6.1","11.4"],"tsc":["CC6.1","CC6.7","CC6.8","CC7.2","CC7.3","CC7.4"]},"agent":{"id":"000","name":"FF_SIEM"},"manager":{"name":"FF_SIEM"},"id":"1692762605.2221522309","full_log":"Aug 23 08:47:23 FFPAFW-1.ffho.org 1,2023/08/23 08:47:23,024301000859,TRAFFIC,end,2562,2023/08/23 08:47:23,172.16.22.46,10.1.0.51,0.0.0.0,0.0.0.0,LAN-ServerFarm,,,incomplete,vsys1,LAN-ZONE,SERVERFARM,ethernet1/23,ethernet1/24,Syslog-server,2023/08/23 08:47:23,2350638,1,65243,1688,0,0,0x19,tcp,allow,66,66,0,1,2023/08/23 08:47:18,0,any,,7269008771584323620,0x0,172.16.0.0-172.31.255.255,10.0.0.0-10.255.255.255,,1,0,aged-out,0,0,0,0,,FFPAFW-1,from-policy,,,0,,0,,N/A,0,0,0,0,fab65291-342b-4eba-83d1-9345f0770599,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2023-08-23T08:47:23.776+05:00,,,unknown,unknown,unknown,1,,,incomplete,no,no,0","predecoder":{"timestamp":"Aug 23 08:47:23","hostname":"FFPAFW-1.ffho.org"},"decoder":{"parent":"paloalto","name":"paloalto"},"data":{"protocol":"tcp","action":"allow","receive_time":"2023/08/23 08:47:23","serial_number":"1234512345","type":"TRAFFIC","content_type":"end","generated_time":"2023/08/23 08:47:23","source_address":"172.16.22.46","destination_address":"10.1.0.51","nat_source_ip":"0.0.0.0","nat_destination_ip":"0.0.0.0","rule_name":"LAN-ServerFarm","application":"incomplete","virtual_system":"vsys1","source_zone":"LAN-ZONE","destination_zone":"SERVERFARM","inbound_interface":"ethernet1/23","outbound_interface":"ethernet1/24","log_action":"Syslog-server","session_id":"2350638","repeat_count":"1","source_port":"65243","destination_port":"1688","nat_source_port":"0","nat_destination_port":"0","flags":"0x19","bytes":"66","bytes_sent":"66","bytes_received":"0","packets":"1","start_time":"2023/08/23 08:47:18","elapsed_time":"0","category":"any","sequence_number":"7269008771584323620","action_flags":"0x0","source_country":"172.16.0.0-172.31.255.255","destination_country":"10.0.0.0-10.255.255.255","packets_sent":"1","packets_received":"0","session_end_reason":"aged-out","device_group_hierarchy_level_1":"0","device_group_hierarchy_level_2":"0","device_group_hierarchy_level_3":"0","device_group_hierarchy_level_4":"0","device_name":"FFPAFW-1","action_source":"from-policy","tunnel_id_imsi":"0","parent_session_id":"0","tunnel_type":"N/A","sctp_association_id":"0","sctp_chunks":"0","sctp_chunks_sent":"0","sctp_chunks_received":"0","rule_uuid":"fab65291-342b-4eba-83d1-9345f0770599","http_2_connection":"0","app_flap_count":"0","high_resolution_timestamp":"2023-08-23T08:47:23.776+05:00","application_subcategory":"unknown","application_category":"unknown","application_technology":"unknown","application_risk":"1","application_saas":"incomplete","application_sanctioned_state":"no"},"location":"1.1.1.1"}
{"timestamp":"2023-08-23T03:50:05.660+0000","rule":{"level":2,"description":"Palo Alto Traffic: Session ended on FFPAFW-1 from 172.16.70.22 to 13.107.42.12. Reason: threat.","id":"64507","firedtimes":253996,"mail":false,"groups":["paloalto"],"gdpr":["IV_35.7.d"],"gpg13":["4.12"],"hipaa":["164.312.b"],"pci_dss":["1.4","10.6.1","11.4"],"tsc":["CC6.1","CC6.7","CC6.8","CC7.2","CC7.3","CC7.4"]},"agent":{"id":"000","name":"FF_SIEM"},"manager":{"name":"FF_SIEM"},"id":"1692762605.2221576848","full_log":"Aug 23 08:47:23 FFPAFW-1.ffho.org 1,2023/08/23 08:47:23,024301000859,TRAFFIC,end,2562,2023/08/23 08:47:23,172.16.70.22,13.107.42.12,119.156.11.111,13.107.42.12,LAN-Internet-VIP-Access,,,ms-onedrive-base,vsys1,LAN-ZONE,PTCL-Zone,ethernet1/23,ethernet1/2,Syslog-server,2023/08/23 08:47:23,2323614,1,50353,443,23747,443,0x40040d,tcp,allow,439,373,66,4,2023/08/23 08:45:53,0,online-storage-and-backup,,7269008771584323646,0x0,172.16.0.0-172.31.255.255,United States,,3,1,threat,0,0,0,0,,FFPAFW-1,from-policy,,,0,,0,,N/A,0,0,0,0,f03f4e73-52c0-4fc8-a42a-b4e49c455446,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2023-08-23T08:47:23.872+05:00,,,file-sharing,saas,client-server,4,\"consume-big-bandwidth,able-to-transfer-file,has-known-vulnerability,prone-to-misuse,pervasive-use,is-saas,is-ip-based-restrictions,no-certifications\",ms-onedrive,ms-onedrive-base,yes,no,0","predecoder":{"timestamp":"Aug 23 08:47:23","hostname":"FFPAFW-1.ffho.org"},"decoder":{"parent":"paloalto","name":"paloalto"},"data":{"protocol":"tcp","action":"allow","receive_time":"2023/08/23 08:47:23","serial_number":"1234512345","type":"TRAFFIC","content_type":"end","generated_time":"2023/08/23 08:47:23","source_address":"172.16.70.22","destination_address":"13.107.42.12","nat_source_ip":"119.156.11.111","nat_destination_ip":"13.107.42.12","rule_name":"LAN-Internet-VIP-Access","application":"ms-onedrive-base","virtual_system":"vsys1","source_zone":"LAN-ZONE","destination_zone":"PTCL-Zone","inbound_interface":"ethernet1/23","outbound_interface":"ethernet1/2","log_action":"Syslog-server","session_id":"2323614","repeat_count":"1","source_port":"50353","destination_port":"443","nat_source_port":"23747","nat_destination_port":"443","flags":"0x40040d","bytes":"439","bytes_sent":"373","bytes_received":"66","packets":"4","start_time":"2023/08/23 08:45:53","elapsed_time":"0","category":"online-storage-and-backup","sequence_number":"7269008771584323646","action_flags":"0x0","source_country":"172.16.0.0-172.31.255.255","destination_country":"United States","packets_sent":"3","packets_received":"1","session_end_reason":"threat","device_group_hierarchy_level_1":"0","device_group_hierarchy_level_2":"0","device_group_hierarchy_level_3":"0","device_group_hierarchy_level_4":"0","device_name":"FFPAFW-1","action_source":"from-policy","tunnel_id_imsi":"0","parent_session_id":"0","tunnel_type":"N/A","sctp_association_id":"0","sctp_chunks":"0","sctp_chunks_sent":"0","sctp_chunks_received":"0","rule_uuid":"f03f4e73-52c0-4fc8-a42a-b4e49c455446","http_2_connection":"0","app_flap_count":"0","high_resolution_timestamp":"2023-08-23T08:47:23.872+05:00","application_subcategory":"file-sharing","application_category":"saas","application_technology":"client-server","application_risk":"4","application_characteristic":"\"consume-big-bandwidth","application_container":"able-to-transfer-file","application_saas":"has-known-vulnerability","application_sanctioned_state":"prone-to-misuse"},"location":"1.1.1.1"}
{"timestamp":"2023-08-23T03:50:03.533+0000","rule":{"level":2,"description":"Palo Alto Traffic: Session ended on FFPAFW-1 from 172.16.70.22 to 68.232.34.200. Reason: tcp-fin.","id":"64507","firedtimes":253685,"mail":false,"groups":["paloalto"],"gdpr":["IV_35.7.d"],"gpg13":["4.12"],"hipaa":["164.312.b"],"pci_dss":["1.4","10.6.1","11.4"],"tsc":["CC6.1","CC6.7","CC6.8","CC7.2","CC7.3","CC7.4"]},"agent":{"id":"000","name":"FF_SIEM"},"manager":{"name":"FF_SIEM"},"id":"1692762603.2220734250","full_log":"Aug 23 08:47:21 FFPAFW-1.ffho.org 1,2023/08/23 08:47:21,024301000859,TRAFFIC,end,2562,2023/08/23 08:47:21,172.16.70.22,68.232.34.200,1.2.3.4,68.232.34.200,LAN-Internet-VIP-Access,,,skype,vsys1,LAN-ZONE,PTCL-Zone,ethernet1/23,ethernet1/2,Syslog-server,2023/08/23 08:47:21,2235380,1,50047,443,18471,443,0x40047a,tcp,allow,13293,3111,10182,49,2023/08/23 08:44:05,181,computer-and-internet-info,,7269008771584323189,0x0,172.16.0.0-172.31.255.255,United States,,24,25,tcp-fin,0,0,0,0,,FFPAFW-1,from-policy,,,0,,0,,N/A,0,0,0,0,f03f4e73-52c0-4fc8-a42a-b4e49c455446,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2023-08-23T08:47:21.747+05:00,,,voip-video,saas,peer-to-peer,5,\"evasive-behavior,consume-big-bandwidth,used-by-malware,able-to-transfer-file,has-known-vulnerability,prone-to-misuse,pervasive-use,is-saas,is-soc1,is-soc2,is-ip-based-restrictions\",,skype,yes,no,0","predecoder":{"timestamp":"Aug 23 08:47:21","hostname":"FFPAFW-1.ffho.org"},"decoder":{"parent":"paloalto","name":"paloalto"},"data":{"protocol":"tcp","action":"allow","receive_time":"2023/08/23 08:47:21","serial_number":"1234512345","type":"TRAFFIC","content_type":"end","generated_time":"2023/08/23 08:47:21","source_address":"172.16.70.22","destination_address":"68.232.34.200","nat_source_ip":"1.1.1.1","nat_destination_ip":"68.232.34.200","rule_name":"LAN-Internet-VIP-Access","application":"skype","virtual_system":"vsys1","source_zone":"LAN-ZONE","destination_zone":"PTCL-Zone","inbound_interface":"ethernet1/23","outbound_interface":"ethernet1/2","log_action":"Syslog-server","session_id":"2235380","repeat_count":"1","source_port":"50047","destination_port":"443","nat_source_port":"18471","nat_destination_port":"443","flags":"0x40047a","bytes":"13293","bytes_sent":"3111","bytes_received":"10182","packets":"49","start_time":"2023/08/23 08:44:05","elapsed_time":"181","category":"computer-and-internet-info","sequence_number":"7269008771584323189","action_flags":"0x0","source_country":"172.16.0.0-172.31.255.255","destination_country":"United States","packets_sent":"24","packets_received":"25","session_end_reason":"tcp-fin","device_group_hierarchy_level_1":"0","device_group_hierarchy_level_2":"0","device_group_hierarchy_level_3":"0","device_group_hierarchy_level_4":"0","device_name":"FFPAFW-1","action_source":"from-policy","tunnel_id_imsi":"0","parent_session_id":"0","tunnel_type":"N/A","sctp_association_id":"0","sctp_chunks":"0","sctp_chunks_sent":"0","sctp_chunks_received":"0","rule_uuid":"f03f4e73-52c0-4fc8-a42a-b4e49c455446","http_2_connection":"0","app_flap_count":"0","high_resolution_timestamp":"2023-08-23T08:47:21.747+05:00","application_subcategory":"voip-video","application_category":"saas","application_technology":"peer-to-peer","application_risk":"5","application_characteristic":"\"evasive-behavior","application_container":"consume-big-bandwidth","application_saas":"used-by-malware","application_sanctioned_state":"able-to-transfer-file"},"location":"1.1.1.1"}