Writing regex for logs

[vinegar]

Hi,
I am using tail input plugin to read a log into fluentd.

logs look like this -
[INFO][0xad2aee71][Dec 12 03:12:46.880][proc:processAbc] This is message text

What will be the regex format for this? Or please point me to resources which will help me.

I wrote something like this-
^\[<level>\]\[<ident>\]\[<time>\]\[<method>\] <message>$

but it doesn't seem proper.

Thanks for your time
-Vin

[Satoshi Tagomori]

Hi Vin,

You should learn Regular expression with capturing.

See: http://www.ruby-doc.org/core-2.0/doc/re_rdoc.html#label-Capturing

2013年12月13日金曜日 18時02分57秒 UTC+9 vinegar:

[vinegar]

Hi Satoshi,

Thanks for reply. I managed to write the format using rubular.com -

  format /\[(?<level>[A-Z]+)\]\[(?<ident>0x[a-f0-9]+)\]\[(?<time>[^ ]* [^ ]* [^ \.]*)[^ ]*\]\[(?<method>[a-z_]+:[a-zA-z_]+)\] (?<message>.*)$/
  time_format %b %d %H:%M:%S

This doesn't throw error when I run fluentd. But I am not getting logs either, need to debug this.

Thanks again,
-Vin

[Kiyoto Tamura]

Hi Vin,

1. Can you paste the configuration file here? That makes it much easier for us to offer help.

2. This might be because your output plugin is buffering. To see if this is the case, try using the copy plugin + stdout plugin like this:

<match **>

  type copy

  <store>

    type <YOUR INTENDED PLUGIN>

    ... # the rest of the command

  </store>

  <store>

    type stdout

  </store>

</match>

If you start seeing events in your stdout, then you know that the data is being sent to the plugin of your choice.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[vinegar]

Hi Kiyoto,

My configuration file-

#es.conf
<source>
  type tail
  path /var/log/syslog
  format syslog
  tag syslog.access
  pos_file /path/to/file/pos/syslog.pos
</source>
<source>
  type tail
  path /path/to/file/mylog.log
  format /\[(?<level>[A-Z]+)\]\[(?<ident>0x[a-f0-9]+)\]\[(?<time>[^ ]* [^ ]* [^ \.]*)[^ ]*\]\[(?<method>[a-z_]+:[a-zA-z_]+)\] (?<message>.*)$/
  time_format %b %d %H:%M:%S
  tag mylog.access
  pos_file /path/to/file/pos/mylog.pos
</source>

<match syslog.access>
  type elasticsearch
  logstash_format true
  flush_interval 5s
</match>
<match mylog.access>
  type elasticsearch
  logstash_format true
  flush_interval 5s
</match>

#On your suggestion I added below
<match **>
  type copy
  <store>
    type file
    path /path/to/file/outmylog.log
    format /\[(?<level>[A-Z]+)\]\[(?<ident>0x[a-f0-9]+)\]\[(?<time>[^ ]* [^ ]* [^ \.]*)[^ ]*\]\[(?<method>[a-z_]+:[a-zA-z_]+)\] (?<message>.*)$/
    time_format %b %d %H:%M:%S
    tag mylog.access
    pos_file /path/to/file/pos/mylog.pos
  </store>
  <store>
    type stdout
  </store>
</match>

I am able to get syslog in kibana, but not mylog. Mylog is simply a log file I copied, it is not dynamically updated. I manually append a message to mylog, to see if fluentd is able to see it.
In outmylof.log I am getting info and error from fluent.

Being new to fluentd, I might have missed something, please let me know.

Thanks

[Masahiro Nakagawa]

> Mylog is simply a log file I copied, it is not dynamically updated.

Currently, in_tail doesn't read existence logs. See following issue:

https://github.com/fluent/fluentd/issues/154

[vinegar]

Thanks Masahiro for the heads-up.

[Sunni Kumar]

Hi,

I have a rails application. But am not able to parse rails log please provide me regular expression for my Rails Logs.
Regular Expression Used by me are here

http://fluentular.herokuapp.com/parse?regexp=Started+%28%3F%3Cmethod%3E[^+]%2B%29+%22%28%3F%3Cpath%3E[^%22]%2B%29%22+for+%28%3F%3Chost%3E[^+]%2B%29+at+%28%3F%3Ctime%3E[^+]%2B+[^+]%2B+[^+]%2B%29\nProcessing+by+%28%3F%3Ccontroller%3E[^\u0023]%2B%29\u0023%28%3F%3Ccontroller_method%3E[^+]%2B%29+as+%28%3F%3Cformat%3E[^+]%2B%3F%29\n%28++Parameters%3A+%28%3F%3Cparameters%3E[^\n]%2B%29\n%29%3F++Rendered+%28%3F%3Ctemplate%3E[^\n]%2B%29+%28%3F%3Cstatus%3E[^+]*%29+%28%3F%3Ccode%3E[^+]*%29+%28%3F%3Chealth%3E[^+]*%29+in+%28%3F%3Cresponse_time%3E[^\n]*%29&input=Started+PUT+%22%2Fattempts%2Fe9bb7e18-3a8e-44b4-b719-201ecd593055%2Fsubmit_problem_attempt%22+for+219.92.107.9+at+2016-01-28+09%3A05%3A57+%2B0000%0D%0AProcessing+by+SkillAttemptsController%23submit_problem_attempt+as+JS%0D%0A++Parameters%3A+{%22utf8%22%3D%3E%22✓%22%2C+%22authenticity_token%22%3D%3E%22t1QluzN%2BOtQUkEZK%2FiZIrpkUa7czgexgRd3ewRLqjEw%3D%22%2C+%22problem_attempt_id%22%3D%3E%226e915f48-de41-417c-b675-9cf972097b56%22%2C+%22answer%22%3D%3E%22{\%22denominator\%22%3A5%2C\%22whole\%22%3A2%2C\%22numerator\%22%3A4}%22%2C+%22user_response%22%3D%3E%22[[\%22col+blue+row-0+col-0+shaded\%22%2C\%22col+blue+row-1+col-0+shaded\%22%2C\%22col+blue+row-2+col-0+shaded\%22%2C\%22col+blue+row-3+col-0+shaded\%22%2C\%22col+blue+row-4+col-0+shaded\%22]%2C[\%22col+blue+row-0+col-0+shaded\%22%2C\%22col+blue+row-1+col-0+shaded\%22%2C\%22col+blue+row-2+col-0+shaded\%22%2C\%22col+blue+row-3+col-0+shaded\%22%2C\%22col+blue+row-4+col-0+shaded\%22]%2C[\%22col+blue+row-0+col-0+shaded\%22%2C\%22col+blue+row-1+col-0+shaded\%22%2C\%22col+blue+row-2+col-0+shaded\%22%2C\%22col+blue+row-3+col-0+shaded\%22]]%22%2C+%22skill_start_date_field%22%3D%3E%22Thu%2C+28+Jan+2016+09%3A05%3A14+GMT%22%2C+%22problem_start_date_field%22%3D%3E%22Thu%2C+28+Jan+2016+09%3A05%3A42+GMT%22%2C+%22problem_seconds_spent_field%22%3D%3E%2213%22%2C+%22problem_attempt_count%22%3D%3E%221%22%2C+%22modernizrProps%22%3D%3E%22{\%22_prefixes\%22%3A[\%22\%22%2C\%22-webkit-\%22%2C\%22-moz-\%22%2C\%22-o-\%22%2C\%22-ms-\%22%2C\%22\%22]%2C\%22_domPrefixes\%22%3A[\%22webkit\%22%2C\%22moz\%22%2C\%22o\%22%2C\%22ms\%22]%2C\%22csstransforms\%22%3Atrue%2C\%22csstransforms3d\%22%3Atrue%2C\%22cssanimations\%22%3Atrue%2C\%22csstransitions\%22%3Atrue}%22%2C+%22id%22%3D%3E%22e9bb7e18-3a8e-44b4-b719-201ecd593055%22}%0D%0A++Rendered+skill_attempts%2Fnew_problem_attempt.js.erb+%286.4ms%29%0D%0ACompleted+200+OK+in+212.6ms+%28Views%3A+5.3ms+%7C+ActiveRecord%3A+63.0ms%29a&time_format=

==================
Started GET "/attempts/e3e5358b-b925-40c0-91bf-00874ba2a3d6/new_problem_attempt?practice_assignment=false&modernizrProps=%7B%22_prefixes%22%3A%5B%22%22%2C%22-webkit-%22%2C%22-moz-%22%2C%22-o-%22%2C%22-ms-%22%2C%22%22%5D%2C%22_domPrefixes%22%3A%5B%22webkit%22%2C%22moz%22%2C%22o%22%2C%22ms%22%5D%2C%22csstransforms%22%3Atrue%2C%22csstransforms3d%22%3Atrue%2C%22cssanimations%22%3Atrue%2C%22csstransitions%22%3Atrue%7D" for 178.89.108.34 at 2016-02-01 10:42:08 +0000
Processing by SkillAttemptsController#new_problem_attempt as */*
  Parameters: {"practice_assignment"=>"false", "modernizrProps"=>"{\"_prefixes\":[\"\",\"-webkit-\",\"-moz-\",\"-o-\",\"-ms-\",\"\"],\"_domPrefixes\":[\"webkit\",\"moz\",\"o\",\"ms\"],\"csstransforms\":true,\"csstransforms3d\":true,\"cssanimations\":true,\"csstransitions\":true}", "id"=>"e3e5358b-b925-40c0-91bf-00874ba2a3d6"}
  Rendered skill_attempts/new_problem_attempt.js.erb (6.8ms)
Completed 200 OK in 141.2ms (Views: 5.5ms | ActiveRecord: 59.8ms)
Started GET "/heartbeat" for 127.0.0.1 at 2016-02-01 10:42:15 +0000
Processing by HeartbeatController#heartbeat as */*
  Rendered text template (0.0ms)
Completed 200 OK in 1.7ms (Views: 0.3ms | ActiveRecord: 0.0ms)
Started GET "/heartbeat" for 127.0.0.1 at 2016-02-01 10:42:15 +0000
Processing by HeartbeatController#heartbeat as */*
  Rendered text template (0.0ms)
Completed 200 OK in 1.5ms (Views: 0.3ms | ActiveRecord: 0.0ms)
Started GET "/heartbeat" for 127.0.0.1 at 2016-02-01 10:42:15 +0000
Processing by HeartbeatController#heartbeat as */*
  Rendered text template (0.0ms)
Completed 200 OK in 1.9ms (Views: 0.4ms | ActiveRecord: 0.0ms)
Started GET "/" for 188.162.185.140 at 2016-02-01 10:42:22 +0000
Processing by StaticController#home as HTML
  Rendered layouts/shared/header/_curriculum_cloud.html.slim (183.7ms)
  Rendered home_pages/v5/_main_banner.html.slim (0.7ms)
  Rendered home_pages/v5/_personalized.html.slim (0.4ms)
  Rendered home_pages/v5/_encourage_child_page.html.slim (0.3ms)
  Rendered home_pages/v5/_track_child_progress.html.slim (0.5ms)
  Rendered home_pages/v4/_teacher_testimonials.html.slim (0.3ms)
  Rendered home_pages/v5/_home.html.slim (187.8ms)
  Rendered devise/registrations/partials/_new_parent_registration_form_popup_wrapper.html.slim (0.0ms)
  Rendered static/home.html.slim within layouts/static (189.1ms)
  Rendered layouts/shared/fonts/_typekit.html.erb (0.0ms)
  Rendered layouts/shared/meta_tags/_twitter_meta_tags.html.erb (0.3ms)
  Rendered layouts/shared/meta_tags/_meta_tags.html.erb (0.5ms)
  Rendered layouts/shared/_airbrake.html.erb (0.0ms)
  Rendered action_traits/_schedule_google_analytics_event.html.slim (0.0ms)
  Rendered layouts/shared/tracking_codes/_google_analytics.html.erb (0.9ms)
  Rendered layouts/shared/tracking_codes/_facebook_retargeting.html.erb (0.0ms)
  Rendered layouts/shared/seo/_canonical_url.html.erb (0.0ms)
  Rendered layouts/shared/partials/_mixpanel_tracking_page_view.html.slim (0.2ms)
  Rendered layouts/shared/tracking_codes/_user_voice.html.erb (0.1ms)
  Rendered layouts/shared/tracking_codes/_optimizely.html.slim (0.0ms)
  Rendered layouts/shared/tracking_codes/_mixpanel.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_segmentio.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_crazy_egg.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_custom_tracking_methods.html.erb (5.8ms)
  Rendered layouts/shared/tracking_codes/_google_remarketing.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_js_libraries.html.slim (9.9ms)
  Rendered layouts/shared/browser_support/not_supported/_ie7_and_older.html.slim (11.6ms)
  Rendered layouts/shared/browser_support/not_supported/_older_browser.html.slim (11.8ms)
  Rendered layouts/shared/exception_handling/_analytics_event_for_mixpanel_exception_handling.html.erb (0.0ms)
  Rendered layouts/shared/header/_guest.html.slim (27.1ms)
  Rendered layouts/shared/header/_header.html.slim (27.9ms)
  Rendered layouts/shared/partials/_banner_notifications.html.slim (0.0ms)
  Rendered layouts/shared/partials/_mixpanel_tracking_page_view.html.slim (0.1ms)
  Rendered layouts/shared/footer/partials/_footer_links_when_logged_out.html.slim (26.4ms)
  Rendered devise/sessions/_login_form_flip_front.html.slim (0.5ms)
  Rendered devise/shared/_oauth_with_buttons.html.slim (1.0ms)
  Rendered devise/shared/_oauth_with_buttons.html.slim (0.9ms)
  Rendered devise/sessions/_login_form_flip_back.html.slim (5.0ms)
  Rendered devise/sessions/_login_form.html.slim (5.8ms)
  Rendered layouts/shared/partials/_login_popup.html.slim (6.0ms)
  Rendered devise/shared/_oauth_with_buttons.html.slim (1.0ms)
  Rendered devise/registrations/partials/_accept_policy.html.erb (0.2ms)
  Rendered layouts/shared/partials/_signup_modal_in_place.html.slim (4.1ms)
  Rendered layouts/shared/partials/_signup_modal.html.slim (4.3ms)
  Rendered devise/registrations/_new.html.erb (0.5ms)
  Rendered layouts/shared/partials/_signup_popup.html.slim (0.8ms)
  Rendered layouts/shared/tracking_codes/_user_voice.html.erb (0.1ms)
  Rendered layouts/shared/tracking_codes/_optimizely.html.slim (0.0ms)
  Rendered layouts/shared/tracking_codes/_mixpanel.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_segmentio.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_crazy_egg.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_custom_tracking_methods.html.erb (0.6ms)
  Rendered layouts/shared/tracking_codes/_google_remarketing.html.erb (0.0ms)
  Rendered layouts/shared/tracking_codes/_js_libraries.html.slim (1.9ms)
  Rendered layouts/shared/footer/_logged_out.html.slim (40.2ms)
  Rendered layouts/shared/footer/_footer.html.slim (40.4ms)
  Rendered layouts/shared/browser_support/supported/_static.html.slim (69.3ms)
Completed 200 OK in 287.3ms (Views: 271.7ms | ActiveRecord: 3.7ms)
===========================================

[Mr. Fiber]

multiline example in in_tail article doesn't work?

http://docs.fluentd.org/articles/in_tail

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.