fluent-plugin-netflow could not parse netflowv9 messages, getting error messages "wrong number of arguments (1 for 0)"

[Koh Yamashita]

Hi 

I've just instaledd td-agent and fluent-plugin-netflow v0.0.3 to integrate ellasticsearch and kibana3 with them, and run into the issue.

Fluent-plugin-netflow cannot parse netflowv9 message from Cisco ASR1K for now.

I attach the pcap file captured the netflow messages.

Please ask me if I can give you more info.

2014-07-13 16:22:48 -0700 [info]: fluent/engine.rb:102:block in configure: adding source type="netflow"

2014-07-13 16:22:48 -0700 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'netflow'

2014-07-13 16:22:48 -0700 [info]: fluent/engine.rb:102:block in configure: adding source type="http"

2014-07-13 16:22:48 -0700 [info]: fluent/engine.rb:118:block in configure: adding match pattern="netflow.*" type="elasticsearch"

2014-07-13 16:22:48 -0700 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'elasticsearch'

2014-07-13 16:22:48 -0700 [debug]: plugin/in_netflow.rb:92:listen: listening netflow socket on 192.168.40.113:5140 with udp

2014-07-13 16:23:16 -0700 [debug]: plugin/in_netflow.rb:73:receive_data: received logs host="10.0.201.31" data="\x00\t\x00\x01x\xEB\xA8dS\xC3\x14\xF6\x00\x00BG\x00\x00\x01\x00\x00\x00\x00D\x01\n\x00\x0F\x00\b\x00\x04\x00\f\x00\x04\x00<\x00\x01\x00\x04\x00\x01\x00\a\x00\x02\x00\v\x00\x02\x008\x00\x06\x00P\x00\x06\x00Q\x00\x06\x009\x00\x06\x000\x00\x01\x00\x01\x00\b\x00\x02\x00\x04\x00\x16\x00\x04\x00\x15\x00\x04"

2014-07-13 16:24:16 -0700 [debug]: plugin/in_netflow.rb:73:receive_data: received logs host="10.0.201.31" data="\x00\t\x00\x01x\xEC\x92\xC4S\xC3\x152\x00\x00BH\x00\x00\x01\x00\x00\x00\x00D\x01\n\x00\x0F\x00\b\x00\x04\x00\f\x00\x04\x00<\x00\x01\x00\x04\x00\x01\x00\a\x00\x02\x00\v\x00\x02\x008\x00\x06\x00P\x00\x06\x00Q\x00\x06\x009\x00\x06\x000\x00\x01\x00\x01\x00\b\x00\x02\x00\x04\x00\x16\x00\x04\x00\x15\x00\x04"

2014-07-13 16:25:16 -0700 [debug]: plugin/in_netflow.rb:73:receive_data: received logs host="10.0.201.31" data="\x00\t\x00\x01x\xED}$S\xC3\x15n\x00\x00BI\x00\x00\x01\x00\x00\x00\x00D\x01\n\x00\x0F\x00\b\x00\x04\x00\f\x00\x04\x00<\x00\x01\x00\x04\x00\x01\x00\a\x00\x02\x00\v\x00\x02\x008\x00\x06\x00P\x00\x06\x00Q\x00\x06\x009\x00\x06\x000\x00\x01\x00\x01\x00\b\x00\x02\x00\x04\x00\x16\x00\x04\x00\x15\x00\x04"

2014-07-13 16:25:47 -0700 [debug]: plugin/in_netflow.rb:73:receive_data: received logs host="10.0.201.31" data="\x00\t\x00\x01x\xED\xF5\x15S\xC3\x15\x8C\x00\x00BJ\x00\x00\x01\x00\x01\n\x00@\xAC\x1Ef\xD3\b\b\b\b\x04\x11\xC1\xB0\x005\xB8x.\x915Y|i\xF6(\xBD\x00|i\xF6(\xBD\x00PW\xA8\x83\x97\x81\x00\x00\x00\x00\x00\x00\x00\x01\xEA\x00\x00\x00\ax\xED\xBA>x\xED\xBAF\x00"

2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:85:rescue in receive_data: "\x00\t\x00\x01x\xED\xF5\x15S\xC3\x15\x8C\x00\x00BJ\x00\x00\x01\x00\x01\n\x00@\xAC\x1Ef\xD3\b\b\b\b\x04\x11\xC1\xB0\x005\xB8x.\x915Y|i\xF6(\xBD\x00|i\xF6(\xBD\x00PW\xA8\x83\x97\x81\x00\x00\x00\x00\x00\x00\x00\x01\xEA\x00\x00\x00\ax\xED\xBA>x\xED\xBAF\x00" error="wrong number of arguments (1 for 0)"

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/base.rb:191:in `to_s'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/parser_netflow.rb:280:in `block in get'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/array.rb:208:in `block in each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/array.rb:208:in `each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/array.rb:208:in `each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/parser_netflow.rb:280:in `collect'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/parser_netflow.rb:280:in `get'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/primitive.rb:111:in `sensible_default'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/base_primitive.rb:142:in `_value'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/primitive.rb:103:in `do_num_bytes'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/struct.rb:250:in `block in sum_num_bytes_below_index'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/struct.rb:247:in `each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/struct.rb:247:in `inject'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/struct.rb:247:in `sum_num_bytes_below_index'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/struct.rb:243:in `sum_num_bytes_for_all_fields'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/struct.rb:141:in `do_num_bytes'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/base.rb:174:in `num_bytes'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/parser_netflow.rb:163:in `block in call'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/array.rb:208:in `block in each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/array.rb:208:in `each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/bindata-2.1.0/lib/bindata/array.rb:208:in `each'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/parser_netflow.rb:63:in `call'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/in_netflow.rb:75:in `receive_data'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/in_netflow.rb:111:in `call'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/in_netflow.rb:111:in `on_readable'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/io.rb:170:in `on_readable'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run_once'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'

  2014-07-13 16:25:47 -0700 [warn]: plugin/in_netflow.rb:111:call: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-netflow-0.0.3/lib/fluent/plugin/in_netflow.rb:64:in `run'

2014-07-13 16:25:48 -0700 [debug]: plugin/in_netflow.rb:73:receive_data: received logs host="10.0.201.31" data="\x00\t\x00\x02x\xED\xF8\xFDS\xC3\x15\x8D\x00\x00BK\x00\x00\x01\x00\x01\n\x00|\xAC\x1Ef\xD3J}\x14T\x04\x06\xE5\x85\x01\xBB\xB8x.\x915Y|i\xF6(\xBD\x00|i\xF6(\xBD\x00PW\xA8\x83\x97\x81\x00\x00\x00\x00\x00\x00\x00\x01\x05\x00\x00\x00\x04x\xED\xBAFx\xED\xBD\xB8\xAC\x1Ef\xD3J}\x14_\x04\x06\xE5\x86\x01\xBB\xB8x.\x915Y|i\xF6(\xBD\x00|i\xF6(\xBD\x00PW\xA8\x83\x97\x81\x00\x00\x00\x00\x00\x00\x00\x01\x05\x00\x00\x00\x04x\xED\xBAFx\xED\xBD\xB0\x00\x00"

2014-07-13 16:25:48 -0700 [warn]: plugin/in_netflow.rb:85:rescue in receive_data: "\x00\t\x00\x02x\xED\xF8\xFDS\xC3\x15\x8D\x00\x00BK\x00\x00\x01\x00\x01\n\x00|\xAC\x1Ef\xD3J}\x14T\x04\x06\xE5\x85\x01\xBB\xB8x.\x915Y|i\xF6(\xBD\x00|i\xF6(\xBD\x00PW\xA8\x83\x97\x81\x00\x00\x00\x00\x00\x00\x00\x01\x05\x00\x00\x00\x04x\xED\xBAFx\xED\xBD\xB8\xAC\x1Ef\xD3J}\x14_\x04\x06\xE5\x86\x01\xBB\xB8x.\x915Y|i\xF6(\xBD\x00|i\xF6(\xBD\x00PW\xA8\x83\x97\x81\x00\x00\x00\x00\x00\x00\x00\x01\x05\x00\x00\x00\x04x\xED\xBAFx\xED\xBD\xB0\x00\x00" error="wrong number of arguments (1 for 0)"

Any comments and advoices will be appreciated.

Regards,

Koh

[Masahiro Nakagawa]

Maybe, lacking definition?

https://github.com/repeatedly/fluent-plugin-netflow/blob/master/lib/fluent/plugin/netflow.yaml

Sorry, I'm now busy so I can't debug it right now.

Does someone check this issue?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Koh Yamashita]

The netflow messages includes fields what the netflow.yaml file defines;

As you can see as below;

    FlowSet 1

        FlowSet Id: Data Template (V9) (0)

        FlowSet Length: 68

        Template (Id = 266, Count = 15)

            Template Id: 266

            Field Count: 15

            Field (1/15): IP_SRC_ADDR

                Type: IP_SRC_ADDR (8)

                Length: 4

            Field (2/15): IP_DST_ADDR

                Type: IP_DST_ADDR (12)

                Length: 4

            Field (3/15): IP_PROTOCOL_VERSION

                Type: IP_PROTOCOL_VERSION (60)

                Length: 1

            Field (4/15): PROTOCOL

                Type: PROTOCOL (4)

                Length: 1

            Field (5/15): L4_SRC_PORT

                Type: L4_SRC_PORT (7)

                Length: 2

            Field (6/15): L4_DST_PORT

                Type: L4_DST_PORT (11)

                Length: 2

            Field (7/15): SRC_MAC

                Type: SRC_MAC (56)

                Length: 6

            Field (8/15): DESTINATION_MAC

                Type: DESTINATION_MAC (80)

                Length: 6

            Field (9/15): SOURCE_MAC

                Type: SOURCE_MAC (81)

                Length: 6

            Field (10/15): DST_MAC

                Type: DST_MAC (57)

                Length: 6

            Field (11/15): FLOW_SAMPLER_ID

                Type: FLOW_SAMPLER_ID (48)

                Length: 1

            Field (12/15): BYTES

                Type: BYTES (1)

                Length: 8

            Field (13/15): PKTS

                Type: PKTS (2)

                Length: 4

            Field (14/15): FIRST_SWITCHED

                Type: FIRST_SWITCHED (22)

                Length: 4

            Field (15/15): LAST_SWITCHED

                Type: LAST_SWITCHED (21)

                Length: 4

Regards,

Koh

[Koh Yamashita]

Hi all,

I could fix the issue, now it is working fine!!

Somehow I modified the follwing code in parser_netflow.rb file, then started working perfectly without any error messages.

< parser_netflow.rb >

      class MacAddr < BinData::Primitive

        array :bytes, :type => :uint8, :initial_length => 6

        def set(val)

          ints = val.split(/:/).collect { |int| int.to_i(16) }

          self.bytes = ints

        end

        def get

++          self.bytes.collect { |byte| format( "%02x" , byte.to_s) }.join(":")

--     self.bytes.collect { |byte| byte.to_s(16) }.join(":")

Regards,

Koh

[Masahiro Nakagawa]

Ah, good catch.

How about 'byte.value.to_s(16)' instead of 'format("%02x" , byte.to_s)'.

I checked BinData and BinData primitives have `value` method to get internal object.

BinData::Uint8#value returns actual Fixnum object which accepts 'to_s(16)'. 

[Koh Yamashita]

Thanks comments!!

I've just verified "'byte.value.to_s(16)'" also works fine!

but as a mac-address, "'byte.value.to_s(16).rjust(2 '0')" looks better ;)

By the way, I found typo in netflow.yaml file;

56:
- :mac_addr
-- - :in_src_max
++ - :in_src_mac
57:
- :mac_addr
-- - :out_dst_max
++ - :out_dst_mac

 

Regards,

Koh

[Masahiro Nakagawa]

Release new version 0.0.4 of fluent-plugin-netflow including your patch.

http://rubygems.org/gems/fluent-plugin-netflow

Thanks!