Feature Request - Netflow.

[Kellan Strong]

Hello Fluentd Devs and other coders,

I don't know if this is the proper place to request a feature or even if I over stepping my bounds but I would like to request a feature for fluentd. If I am wrong then please let me know, but I would like to see netflow in fluentd. As either a plugin or a feature.

The reason I am asking this is because Logstash supports a codec called netflow. This allows monitoring of network traffic and allowing people to view network traffic as logs.

Another Engineer and I have played around with Fluentd trying to get this to work but logstash already has something for it. Below is the basic config of Logstash that I am using in DevOps with Fluentd. Both work great together and I have no problem using them both. However, I would like to keep my servers less complex, and not run Fluentd and Logstash together.

input {
    udp {
        type => "netflow"
        host => "127.0.0.1"
        port => 9995
        codec => "netflow"
    }
}

output {
    elasticsearch {
        host => "127.0.0.1"
        index => "logstash-%{+YYYY.MM.dd.HH}"
    }
}

This works with softflowd to produce traffic on the network in the form of logs. I am not a Ruby coder so I wouldn't even know where to start with trying to convert this to a plugin for fluentd. Also, If anyone has actually done netflow with fluentd before let me know because I am interested.


Thanks,

[Kiyoto Tamura]

Hi Kellen,

Thanks for asking. I will look into it.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector for high-volume data streams

[Masahiro Nakagawa]

Maybe, porting logstash's netflow codes is easy as fluent-plugin-netflow or something.

But I have several questions because I don't know the Netflow.

1. Does Netflow payload contain several events in one flow? I checked Logstash code and found `flowset.records` code.

2. Does Netflow support only UDP or several protocols?

  - if so, what is the separator between each payload?

3. Do you have a sample logs?

If 1. is true, we may need to improve the TextParser API to accept multi events in one line.

[Kellan Strong]

This is what it looks like in Kibana.

It can do both UDP and TCP. But I have it as UDP for faster streaming.

From what I understand, its doing multiple events in one flow but I am not sure if it actually is doing that and its just not one large event. Since softflowd has no long I cannot view it.

This is what it is in JSON.

{
  "_index": "logstash-2014.06.16.15",
  "_type": "netflow",
  "_id": "_DV9u_1JS0mt_zpFV9oJ0Q",
  "_score": null,
  "_source": {
    "@timestamp": "2014-06-16T15:57:01.000Z",
    "netflow": {
      "version": "9",
      "flow_seq_num": "11989604",
      "flowset_id": "1024",
      "ipv4_src_addr": "127.0.0.1",
      "ipv4_dst_addr": "127.0.0.1",
      "last_switched": "2014-06-16T14:56:56.999Z",
      "first_switched": "2014-06-16T14:56:56.999Z",
      "in_bytes": "534",
      "in_pkts": "4",
      "input_snmp": "0",
      "output_snmp": "0",
      "l4_src_port": "9200",
      "l4_dst_port": "58408",
      "protocol": "6",
      "tcp_flags": "27",
      "ip_protocol_version": "4"
    },
    "@version": "1",
    "type": "netflow",
    "host": "127.0.0.1"
  },
  "sort": [
    1402934221000,
    1402934221000
  ]
}

Please let me know if there is anything else I can help with.

[Masahiro Nakagawa]

I want to know raw log, not parsed result.

I don't have Netflow so need raw log to check the parser.

[Kellan Strong]

Oh sorry,

ACTIVE seq:12693098 [10.42.0.55]:37218 <> [10.42.0.55]:9200 proto:6 octets>:410 packets>:4 octets<:0 packets<:0 start:2014-06-19T20:40:10.009 finish:2014-06-19T20:40:10.011 tcp>:1a tcp<:00 flowlabel>:00000000 flowlabel<:00000000
EXPIRY EVENT for flow 12693098 in 3600 seconds

[Kellan Strong]

Additionally, Fluentd is picking this up from syslog after increasing the debug level:

softflowd[31933]: ADD FLOW seq:12695440 [127.0.0.1]:50266 <> [127.0.0.1]:9200 proto:6

[Mr Tim F]

I'm interested in this as well...Kellan, what's the environment you are in and what are you trying to do once fluentd can input/output the data?

Are you trying to ingest/parse/search/visualize?

[Kellan Strong]

I'm doing this in a dev environment on ubuntu 12.04, I think that's what you are asking?, and if it works well move it to our production environment after CI/QA. I am trying to have visualization for security purposes for traffic between all our hosts. And I was going to build a python script(I am probably going to replace it with the Zabbix plugin for fluentd but I have yet to test it) to query elasticsearch and send it to our monitoring system (Zabbix) to alert to our NOC for any kinda of weird behavior. I really just want source and destination ips and packets. One of our Network Engineers is really interested in this data as well.

[Masahiro Nakagawa]

I wrote fluent-plugin-netflow.

https://github.com/repeatedly/fluent-plugin-netflow

Ported Netflow parser can't parse your pasted log so Netflow log seems to have more information.

So for finishing the implementation, test with Netflow is needed.

Please try above.

[Kellan Strong]

Wow. Awesome. I will definitely test this thoroughly. I will give you a report on it hopefully next week.

[Tim Fong]

Very cool!  Has this been tested in a production environment.  Curious if it's Cisco gear. 

You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/V5j1Znx4gjg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from Gmail Mobile

[Tim Fong]

Kellan that's very interesting.   I was wondering if you checked out influxdb. I am trying to set it up for a similar use case of network monitoring of syslog and netflow.ore from a dashboard NOC than search. 

You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/V5j1Znx4gjg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from Gmail Mobile

[Kellan Strong]

Here it is in json off of Kibana. It works right out of the box almost. I just have to play around with the elasticsearch plugin a bit to give it a different _type so I can get this away from normal syslog messages.

{
  "_index": "logstash-2014.06.24.17",
  "_type": "fluentd",
  "_id": "2L7OP0opQpOX4QHSForpSQ",
  "_score": null,
  "_source": {
    "version": 9,
    "flow_seq_num": 13441864,
    "flowset_id": 1024,
    "ipv4_src_addr": "10.42.0.55",
    "ipv4_dst_addr": "10.42.0.214",
    "last_switched": "2014-06-24T17:06:54.999Z",
    "first_switched": "2014-06-24T17:06:54.999Z",
    "in_bytes": 554,
    "in_pkts": 6,
    "input_snmp": 0,
    "output_snmp": 0,
    "l4_src_port": 53130,
    "l4_dst_port": 10051,
    "protocol": 6,
    "tcp_flags": 27,
    "ip_protocol_version": 4,
    "host": "127.0.0.1",
    "@timestamp": "2014-06-24T10:12:01-07:00"
  },
  "sort": [
    1403629921000,
    1403629921000
  ]
}

[Kellan Strong]

While reading the logs I got this:

No matching template for flow id 1024

about 15 times then it disappeared with no other problems nor are there any problems when it first boots up.

[Masahiro Nakagawa]

It occurred at this point.

https://github.com/repeatedly/fluent-plugin-netflow/blob/0fb4639a5498fb04d76d085e46c50169c9677da8/lib/fluent/plugin/parser_netflow.rb#L151

Maybe, logstash also generates same log.

https://github.com/elasticsearch/logstash/blob/65af91d0158725fd18dc09897c6d73839efa47d0/lib/logstash/codecs/netflow.rb#L175

I don't understand why this warning occurred but it is warning, not error on original parser. 

So it seems not critical I think.