Nxlog and Fluentd

[Kellan Strong]

Hello all,

I have been for the life of me trying to to use NXLOG to send access.logs to my fluent server over port 5140. I recently switched over from logstash to fluent and I am liking it a lot more. On the two hosts I have opened the port 5140 but still there is nothing in elasticsearch for my access log. All help is appreciated. Here is my current config for fluent that I have been testing out.

  <source>
    type syslog
    port 5140
    tag apache.access
    format apache2
  </source>
  <source>
    type forward
  </source>
  <match **>
    type elasticsearch
    logstash_format true
    flush_interval 5s
  </match>
  <source>
    type tail
    path /var/local/logstash/*.log
    pos_file /var/log/td-agent/syslog.log.pos
    tag system.local
    format /^(?<time>[^ ]* [^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?[^\:]*\: *(?<message>.*)$/
    time_format %b %d %H:%M:%S
  </source>
  <match **>
    type elasticsearch
    logstash_format true
    flush_interval 5s
  </match>

The nxlog config

########################################
# Global directives                    #
########################################
User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

########################################
# Modules                              #
########################################
<Extension json>
    Module xm_json
</Extension>

<Input in>
    Module im_file
    File "/var/log/apache2/access.log"
    SavePos  TRUE
</Input>

<Output out>
    Module      om_udp
    Host        192.168.1.55
    Port        5140
</Output>

########################################
# Routes                               #
########################################
<Route 1>
    Path        in => out
</Route>

[Masahiro Nakagawa]

Hi Kallen,

Thank you for trying Fluentd :)

> still there is nothing in elasticsearch for my access log 

From this document http://docs.fluentd.org/articles/windows,

Basic configuration seems correct but

I don't know how customize the configuration for apache logs.

Could you show me the error log of /var/log/td-agent/td-agent.log?

I assume `pattern not match` occurred in syslog input plugin.

Kiyoto:

Could you check nxlog configuration for apache logs?

Thanks,

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Masahiro Nakagawa]

If you show me the log example which is generated by NXLOG,

we can provide correct configuration and adding it to docs.fluentd.org.

For example, http://docs.fluentd.org/articles/windows document 

assumes NXLOG sends following like format to Fluentd's in_syslog.

<1>Feb 20 00:00:00 portly fluentd[11111]: [notice] Windows is awesome

We can support various syslog format is configuring format option or

using fluent-plugin-parser to parse `message` filed.

I wait your reply :)

[Kellan Strong]

今日は

I was using that(http://docs.fluentd.org/articles/windows) as a base for how to get it to work. Since the other engineer who actually set up nxlog to work with logstash didn't actually get it to work so internally I have no help with this tool. It seems to much of a hassle to do it like the rest of the logs because those are on a mount that is mounted on our syslog host and mounted on my fluent host.

Thing is there really is nothing in the logs for td-agent, i get a lot of patterns not match for items that actually show up in the logs but that's a different matter. I did get this after I restarted, but only once in nxlog. Restarting it again does not show this.

2014-04-17 14:54:03 ERROR om_udp apr_socket_send failed;Connection refused
2014-04-17 15:00:03 ERROR om_udp apr_socket_send failed;Connection refused
2014-04-17 15:00:06 ERROR om_udp apr_socket_send failed;Connection refused
2014-04-17 15:02:36 ERROR om_udp apr_socket_send failed;Connection refused
2014-04-17 14:54:03 ERROR om_udp apr_socket_send failed;Connection refused

This may be a problem with my configuration or even how I set up iptables rules on both hosts.

Thank you for your help.

[Serge Zz]

     Hi Kellan,

  NXLog is great and fast tool and I've sucessfully married it with fluentd.  My architecture is as follows: nxlog agents tailing apache access logs on web cluster machines - multiple machines sending their access logs to a common place - a log collector running another nxlog instance, fluentd, Elasticsearch, nginx and Kibana.
Here is critical part of /etc/nxlog/nxlog.conf  on apache hosts (Ubuntu 12.04) :
<Input access_log>
    Module      im_file
    File        "/var/log/apache2/mywebvhohst.access.log"
    Exec    if $raw_event =~ /^(\S+) (\S+) (\S+) \[([^\]]+)\] \"(\S+) (.+) HTTP.\d\.\d\" (\d+) (\d+) \"([^\"]+)\" \"([^\"]+)\"/\
                { \
                  $Hostname = $1; \
                  if $3 != '-' $AccountName = $3; \
                  $EventTime = parsedate($4); \
                  $HTTPMethod = $5; \
                  $HTTPURL = $6; \
                  $HTTPResponseStatus = $7; \
                  $FileSize = $8; \
                  $HTTPReferer = $9; \
                  $HTTPUserAgent = $10; \
                }
</Input>
<Output out_udp>
    Module      om_udp
    Port        514
    Host        10.1.1.9

</Output>
########################################
# Routes                               #
########################################

<Route apache>
    Path        access_log => out_udp
</Route>

 Now here is nxlog config at the log collector machine (10.1.1.9):

# The buffer needed to NOT lose events when fluentd restarts
<Processor buffer_udp>

    Module      pm_buffer
    # 1Mb buffer
    MaxSize 1024

    Type Mem
    # warn at 512k
    WarnLimit 512

</Processor>
<Input in1_udp>
    Module      im_udp
    Host        0.0.0.0
    Port        514
     Exec    if $raw_event =~ /^(\S+) (\S+) (\S+) \[([^\]]+)\] \"(\S+) (.+) HTTP.\d\.\d\" (\d+) (\d+) \"([^\"]+)\" \"([^\"]+)\"/\
                { \
                  $Hostname = $1; \
                  if $3 != '-' $AccountName = $3; \
                  $EventTime = parsedate($4); \
                  $HTTPMethod = $5; \
                  $HTTPURL = $6; \
                  $HTTPResponseStatus = $7; \
                  $FileSize = $8; \
                  $HTTPReferer = $9; \
                  $HTTPUserAgent = $10; \
                }  else  drop();
 #  sometime there are misformed lines (not too many though), so we drop them
</Input>

# looks like there is no reliable way to feed it directly to fluentd, so we write all our apache access logs to a common  file  which will be scanned/tailed by fluentd
<Output fileout1>
    Module      om_file
    File        "/var/log/apache/apache-access.log"
    # we convert it to JSON on the fly - the nxlog is great for this and very fast - pure C :)
    Exec        $raw_event = to_json();

</Output>
########################################
# Routes                               #
########################################

<Route routeout>
     Path       in1_udp => buffer_udp => fileout1
 </Route>

Fluentd  at log collector (Ubuntu again) /etc/td-agent/td-agent.conf :
<source>
 #  here I wish  Fluentd developers would create another plugin similar to this one but  to be udp/tcp enabled
  type tail
  path /var/log/apache/apache-access.log
  pos_file /var/log/td-agent/httpd-access.log.pos
  # our logs are already in JSON - thanks to nxlog, so this will make flentd running easier
  format json
  tag apache.access
</source>

# and finally we feed it to Elasticsearch
<match apache.**>
  type elasticsearch
  logstash_format true
  logstash_prefix adcenter
  flush_interval 10s # for testing
</match>

   I had nightmares to get Logstash and its agents working proper way, on Ubuntu it was working horrible that why I switched to nxlog + Fluentd . How to configure Elasticsearch and Kibana - there is plenty of documents especially for Logstash + Elasticsearch + Kibana - you throw the Logstash away and do the rest :)

 Fluentd seem to work fine, the only thing is I wish there was an input plugin similar to 'in_tail' - same functionality and parameters,  same supported formats , etc,  but tcp/udp enabled  so I wouldn't need to create\tail that /var/log/apache/apache-access.log file. I think this would speed things up  saving on disk I/O. Unfortunately I'm not very strong in programming to create such a plugin myself.

 Hope this will help.

[Masahiro Nakagawa]

Hi Serge,

Thank you for sharing the configuration.

Another guy uses fluentd's in_http and nxlog's om_http.

This approach is also good.

https://gist.github.com/kan6004/8fc90bec059515e081bf

We are now researching the best configuration to receive logs from nxlog.

> I wish there was an input plugin similar to 'in_tail' - same functionality and parameters,  same supported formats , etc,  but tcp/udp enabled

I'm not sure nxlog's output. But if one output sends arbitrary JSON over TCP,

you can use in_forward to receive logs from nxlog.

["tag string", time, {...JSON...}]

http://docs.fluentd.org/articles/in_forward#protocol

Masahiro

--