parsing with netflow having "pattern not match"
Sam Iland
<source>
@type netflow
tag netflow.event
port 5145
cache_ttl 6000
versions [5, 9]
definitions /etc/td-agent/plugin/netflow_fields.yaml
</source>
<source>
@type tail
path /var/log/td-agent/netflow_20180329.log
pos_file /var/log/td-agent/netflow.pos.log
read_from_head true
<parse>
@type json
</parse>
</source>
Netflow data:
2018-03-31T17:33:23+08:00 netflow.event {“version”:5,”uptime”:34846000,”flow_records”:30,”flow_seq_num”:150,”engine_type”:0,”engine_id”:0,”sampling_algorithm”:0,”sampling_interval”:0,”ipv4_src_addr”:”192.168.10.10”,”ipv4_dst_addr”:”192.168.10.254”,”ipv4_next_hop”:”0.0.0.0”,”input_snmp”:0,”output_snmp”:2,”in_pkts”:1,”in_bytes”:52,”first_switched”:”2018-03-29T09:15:07.966Z”,”last_switched”:”2018-03-29T09:15:07.966Z”,”14_src_port”:80,”14_dst_port”:63569,”tcp_flags”:16,”protocol”:6,”src_tos”:0,”src_as”:0,”dst_as”:0,”src_mask”:32,”dst_mask”:24,”host”:”192.168.0.10”
Error log:
Error Type: 2018-03-31 14:25:46 +0800 [warn]: #0 pattern not match: “2018-03-31T17: 15:23+08:00\tnetflow.event\t{\”version\”:5,\”uptime\”:34846000,\”flow_records\”:30,\”flow_seq_num\”:150,\”engine_type\”:0,\”engine_id\”:0,\”sampling_algorithm\”:0,\sampling_interval\”:0,\”ipv4_src_addr\”:\”192.168.10.10\”,\”ipv4_dst_addr\”:\”192.168.10.254\”,\”ipv4_next_hop\”:\”0.0.0.0\”,\”input_snmp\”:0,\”output_snmp\”:2,\,\”in_pkts\”:1,\”in_bytes\”:52,\”first_switched\”:\2018-03-31T09:15:07.9662\”,\”last_switched\”:\2018-03-31T09:15:07.9662\”,\”14_src_port\”:88,\”14_dst_port\”:63569,\”tcp_flags\”:16,\”protocol\”:6,\”src_tps\”:0,\”src_as\”:0,\”dst_as\”:0,\”src_mask\”:32,\”dst_mask\”:24,\”host\”:\”192.168.0.10\”}"`
I try changing with regexp, nginx for testing, but getting same error... I believe my expression is wrong...
I want to have fluentd to generate a file with line breaker to make it more reader friendly, like below: (sample from Fluentd JSON parse).
From Log
{"time":1362020400,"host":"192.168.0.1","size":777,"method":"PUT"}
Parsed to:
time:
1362020400 (2013-02-28 12:00:00 +0900)
record:
{
"host" : "192.168.0.1",
"size" : 777,
"method": "PUT",
}
Thank you in advance.
Mr. Fiber
--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sam Iland
2018-03-31T17:33:23+08:00 netflow.event {“version”:5,”uptime”:34846000,”flow_records”:30,”flow_seq_num”:150,”engine_type”:0,”engine_id”:0,”sampling_algorithm”:0,”sampling_interval”:0,”ipv4_src_addr”:”192.168.10.10”,”ipv4_dst_addr”:”192.168.10.254”,”ipv4_next_hop”:”0.0.0.0”,”input_snmp”:0,”output_snmp”:2,”in_pkts”:1,”in_bytes”:52,”first_switched”:”2018-03-29T09:15:07.966Z”,”last_switched”:”2018-03-29T09:15:07.966Z”,”14_src_port”:80,”14_dst_port”:63569,”tcp_flags”:16,”protocol”:6,”src_tos”:0,”src_as”:0,”dst_as”:0,”src_mask”:32,”dst_mask”:24,”host”:”192.168.0.10”
Mr. Fiber
--