Logs missing at EFK

[Sunni Kumar]

Hi All, 

Am using Elasticsearch, Fluentd & Kibana for centralized logging. But i have been noticed some of log lines has been missed. I am going to share my server & client details below kindly check whats wrong with my configuration.

==================================================

SERVER SIDE CONFIG

==================================================

#####

<source>

  type http

  port 8888

</source>

<source>

  type secure_forward

  secure no

  shared_key sUlWoNk

  self_hostname ${hostname}

# cert_auto_generate yes

  buffer_type file

  buffer_path /var/log/td-agent/secure_buffer

  flush_interval 30s

  buffer_queue_limit 245

  buffer_chunk_limit 64m

  port 24224

</source>

<match All.Nginx.Error*>

type record_reformer

tag web.error.${hostname}

</match>

<match All.Nginx.Access*>

type record_reformer

tag web.access.${hostname}

</match>

<match All.Rails.Production*>

type record_reformer

tag web.production.${hostname}

</match>

<match All.Unicorn.Stdout*>

type record_reformer

tag web.unicron.stdout.${hostname}

</match>

<match All.Unicorn.Stderr*>

type record_reformer

tag web.unicron.stderr.${hostname}

</match>

<match All.Logstash.Production*>

type record_reformer

tag web.logstash_production.${hostname}

</match>

<match All.Logstash.Production*>

type record_reformer

tag lc.logstash_production.${hostname}

</match>

<match All.Nginx.Access*>

type record_reformer

tag lc.access.${hostname}

</match>

<match All.Unicorn.*>

type record_reformer

tag lc.unicron..${hostname}

</match>

<match All.Staging.Error*>

type record_reformer

tag sync.error.${hostname}

</match>

<match All.Staging.Access*>

type record_reformer

tag sync.access.${hostname}

</match>

<match All.Rails.Sync*>

type record_reformer

tag sync.sync.${hostname}

</match>

<match All.Unicorn.Stdout*>

type record_reformer

tag sync.unicron.stdout.${hostname}

</match>

<match All.Unicorn.Stderr*>

type record_reformer

tag sync.unicron.stderr.${hostname}

</match>

<match All.Sync*>

type record_reformer

tag sync.sync.${hostname}

</match>

<match All.Logstash.Sync*>

type record_reformer

tag web.logstash_sync.${hostname}

</match>

<match *.**>

  type forest

  #type copy

  #type stdout

  remove_prefix All

  subtype copy

  <template>

    <store>

      type elasticsearch

      host 127.0.0.1

      port 9200

      flush_interval 30s

      time_key end_stamp

      logstash_format true

                include_tag_key true

                tag_key @log_name

      utc_index false

                #time_format %Y-%m-%d %H:%M:%S

    </store>

#<store>

#type stdout

#</store>

  </template>

</match>

<source>

  type debug_agent

  bind 127.0.0.1

  port 24230

</source>

==================================================

CLIENT SIDE CONFIG

==================================================

####

<source>

  type tail

  path /var/log/nginx/access.log

  pos_file /var/log/td-agent/nginx.access.pos

  tag web.access.log

  refresh_interval 5

  format /^(?<remote>[^-]*)([ -]*) \[(?<time>[^\]]*)\] \"(?<method>[^ ]*) (?<path>[^ ]*) (?<protocol>[^\"]*)\" (?<code>[^ ]*) (?<size>[^ ]*) \"(?<request_url>[^\"]*)\" \"(?<agent>[^\"]*)\" \"(?<x_forwarded>[^\"]*)\"/

  time_format %d/%b/%Y:%H:%M:%S %z

  types code:integer,size:integer

</source>

<source>

  type tail

  path /var/log/nginx/error.log

  pos_file /var/log/td-agent/nginx-error.pos

  tag web.error.log

  refresh_interval 5

  format /^(?<time>[^\[]*) \[(?<error-level>[^\]]*)] (?<pid>[^#]*)#(?<tid>[^:]*): \*(?<error_message>[^(]*) \((?<code>[^\:]*)\: (?<message>[^,]*), client: (?<client>[^,]*), server: (?<server>[^,]*), request: \"(?<method>[^ ]*) (?<request>[^ ]*) (?<protocol>[^\"]*)\", upstream: \"(?<upstream>[^\"]*)\", host: \"(?<host>[^\"]*)\"/

  types code:integer,size:integer

</source>

<source>

  type tail

  path /var/application/studypad/current/log/unicorn.stdout.log

  pos_file /var/log/td-agent/unicorn.stdout.log

  tag web.unicorn.stdout.log

  refresh_interval 5

  format none

  types code:integer,size:integer

</source>

<source>

  type tail

  path /var/application/studypad/current/log/unicorn.stderr.log

  pos_file /var/log/td-agent/unicorn.stderr.log

  tag web.unicorn.stderr.log

  refresh_interval 5

  format /^(?<remote>[^-]*)([ -]*) \[(?<time>[^\]]*)\] "(?<method>[(POST|GET)]*) (?<path>[^ ]*) (?<protocol>[^\"]*)\" (?<code>[^ ]*) - (?<size>[^ ]*)/

  format none

  types code:integer,size:integer

</source>

<source>

  type tail

  path /var/application/studypad/current/log/logstash_production.log

  pos_file /var/log/td-agent/logstash_production.log

  tag web.logstash_production.log

  refresh_interval 5

  format json

</source>

<match web.error.*>

  type record_modifier

  gen_host "#{Socket.gethostname}"

  tag "All.Nginx.Error.#{Socket.gethostname}"

  num_threads 8

</match>

<match web.access.*>

  type record_modifier

  gen_host "#{Socket.gethostname}"

  tag "All.Nginx.Access.#{Socket.gethostname}"

  num_threads 8

</match>

<match web.production.*>

  type record_modifier

  gen_host "#{Socket.gethostname}"

  tag "All.Rails.Production.#{Socket.gethostname}"

  num_threads 8

</match>

<match web.unicorn.stderr.*>

  type record_modifier

  gen_host "#{Socket.gethostname}"

  tag "All.Unicorn.Stderr.#{Socket.gethostname}"

  num_threads 8

</match>

<match web.unicorn.stdout.*>

  type record_modifier

  gen_host "#{Socket.gethostname}"

  tag "All.Unicorn.Stdout.#{Socket.gethostname}"

  num_threads 8

</match>

<match web.logstash_production.*>

  type record_modifier

  gen_host "#{Socket.gethostname}"

  tag "All.Logstash.Production.#{Socket.gethostname}"

  num_threads 8

</match>

<match All.**>

  type secure_forward

  secure no

  shared_key sHHSUlWoNk

  self_hostname "#{Socket.gethostname}"

#  cert_auto_generate no

  keepalive 3600

  buffer_type file

  buffer_path /var/log/td-agent/log_buffer/fluent.*.buffer

  buffer_chunk_limit 1m

  buffer_queue_limit 2048

  flush_interval 30s

  retry_limit 10

  retry_wait 100s

 <server>

  host xxxxx

  port 24224

 </server>

</match>

<match debug.**>

  type stdout

</match>

<source>

  type forward

</source>

<source>

  type http

  port 8888

</source>

## live debugging agent

<source>

  type debug_agent

  bind 127.0.0.1

  port 24230

</source>

=======================================

Also Explain me what is the meaning of this  line mentioned in td-agent.log file 

======================================

[ubuntu@/var/log/td-agent]$ sudo tailf td-agent.log

2017-10-04 11:45:45 +0000 [warn]: pattern not match: "2017/10/04 11:45:45 [warn] 16435#16435: *18182 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000024, client: 10.0.0.244, server: xxxxx, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:46:00 +0000 [warn]: pattern not match: "2017/10/04 11:46:00 [warn] 16434#16434: *18245 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000025, client: 10.0.0.244, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:47:07 +0000 [warn]: pattern not match: "2017/10/04 11:47:07 [warn] 16434#16434: *18453 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000026, client: 10.0.3.174, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:48:12 +0000 [warn]: pattern not match: "2017/10/04 11:48:12 [warn] 16434#16434: *18919 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000027, client: 10.0.3.174, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:48:14 +0000 [warn]: pattern not match: "2017/10/04 11:48:14 [warn] 16434#16434: *18931 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000028, client: 10.0.0.244, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:53:30 +0000 [warn]: pattern not match: "2017/10/04 11:53:30 [warn] 16435#16435: *20714 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000029, client: 10.0.0.244, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:56:22 +0000 [warn]: pattern not match: "2017/10/04 11:56:22 [warn] 16435#16435: *22075 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000030, client: 10.0.0.244, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:56:25 +0000 [warn]: pattern not match: "2017/10/04 11:56:25 [warn] 16435#16435: *22075 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000031, client: 10.0.0.244, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:56:51 +0000 [warn]: pattern not match: "2017/10/04 11:56:51 [warn] 16434#16434: *22294 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000032, client: 10.0.3.174, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

2017-10-04 11:56:51 +0000 [warn]: pattern not match: "2017/10/04 11:56:51 [warn] 16435#16435: *22258 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000033, client: 10.0.0.244, server: www. xxxxx.com, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

[Mr. Fiber]

> 2017-10-04 11:45:45 +0000 [warn]: pattern not match: "2017/10/04 11:45:45 [warn] 16435#16435: *18182 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000024, client: 10.0.0.244, server: xxxxx, request: \"POST /webhooks/closeio/events.json HTTP/1.1\", host: \"www. xxxxx.com\""

`pattern not match` means your regexp pattern doesn't match incoming logs.

So if your missing logs are shown in td-agent.log, it measn your in_tail's regexp pattern is wrong for your entire logs.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.