Any log viewer?

[Keisuke Nishida]

Hi,

I'm starting to collect my application log using fluentd,
storing the log into MongoDB or something similar.

Is there any convenient tools (or service) for viewing,
monitoring, and analyzing the collected log?

I'm thinking of something like this. Each entry of our log has
at least a timestamp, tags, and a message. I'd like to view
the log with a specific time range, filtering by a set of tags
or searching the messages. It would be better if the viewer
had a graphical interface with basic functionality of graphing,
like the daily volume of log entries.

I'm also seeking a tool for monitoring my application statistics
and visualizing the activity as a set of graphs. We could set
up a monitoring tool like Munin, writing plugins for our own.
Still, it is some effort to maintain counters by ourselves.

We could instead send application events as log entries and
compute the statistics on the server side, not in the application.
If there were a common way of expressing application events,
such as user logins or posting files, as JSON objects, it might
be easier to visualize the application activity; just post events
with fluentd, and the server will do the rest.

Any idea?

Thanks,

[Kazuki Ohta]

Hi, Keisuke

> Is there any convenient tools (or service) for viewing,
> monitoring, and analyzing the collected log?

That's the one we must have, but unfortunately we don't have it.

> I'm also seeking a tool for monitoring my application statistics
> and visualizing the activity as a set of graphs.

Here at Treasure Data, Inc., we're installing Fluentd into all nodes
and collecting system metrics like current queue depth. Then, we
developed the plugin to post those values to librato metrics (https://
metrics.librato.com/). Another possible candidate was Graphite (http://
graphite.wikidot.com/), but we didn't want to host it by ourselves.
The plugins we're using will soon be included into Fluentd, or open
sourced soon.

Thanks - K

[Keisuke Nishida]

Hi Kazuki,

> Here at Treasure Data, Inc., we're installing Fluentd into all nodes
> and collecting system metrics like current queue depth. Then, we
> developed the plugin to post those values to librato metrics (https://
> metrics.librato.com/). Another possible candidate was Graphite (http://
> graphite.wikidot.com/), but we didn't want to host it by ourselves.
> The plugins we're using will soon be included into Fluentd, or open
> sourced soon.

Looks nice. I'll try them.

Thanks!

2012/2/17 Kazuki Ohta <kazuk...@gmail.com>:

[Keisuke Nishida]

Hi there,

> > Is there any convenient tools (or service) for viewing, 
> > monitoring, and analyzing the collected log? 
> That's the one we must have, but unfortunately we don't have it. 

I've found that a simple plugin like this allows me to use Loggly (loggly.com)

as a log viewer:

--- out_loggly.rb

module Fluent

class LogglyOutput < Fluent::BufferedOutput

  Fluent::Plugin.register_output('loggly', self)

  def initialize

    super

    require 'socket'

  end

  config_param :host, :string, :default => "logs.loggly.com"

  config_param :port, :integer

  def format(tag, time, record)

    "#{record.to_json}\n"

  end

  def write(chunk)

    sock = TCPSocket.open(@host, @port)

    begin

      chunk.write_to(sock)

    rescue

      sock.close

    end

  end

end

end

----

---- fluent.conf

<match app.*>

  type loggly

  port YOUR-PORT-HERE

</match>

----

This might be a light way of indexing, searching, and visualizing your logs as follows:

http://loggly.com/blog/2011/06/on-the-way-to-impressive/

(Note: use "Syslog TCP w/Strip" input, JSON enabled, instead of HTTP input.)

Cheers,

[Keisuke Nishida]

Hi there,

> > Is there any convenient tools (or service) for viewing, 
> > monitoring, and analyzing the collected log? 
> That's the one we must have, but unfortunately we don't have it. 

[Linc Deng]

seems loggly is not free to use.

--
Linc

[Keisuke Nishida]

Right.  If you prefer free software, logstash (logstash.net) may help.

[Keisuke Nishida]

I wrote:

> I've found that a simple plugin like this allows me to use Loggly (loggly.com) as a log viewer:

Just another way I'm trying to do these days..

We could use papertrail (papertrailapp.com) as a log viewer.  Although papertrail doesn't

have facilities for visualization by itself, it can easily be combined with Librato Metrics

(metrics.librato.com) for counting logs that match saved patterns.

So, all I need is an "out_syslog" plugin.  Is there anyone working on that?

thanks,