How to replace syslog-ng with fluentd?

[vmi...@gmail.com]

Hi,

I'm new to fluentd and would like some suggestions on how best to introduce it to my existing setup.

My current setup has a central log server running debian + syslog-ng on collecting logs from all LAN clients, nothing too elaborate but it gets the job done.

But now I'm interested in using fluentd + elasticsearch + kibana to do more, so what's the best way to proceed? Suggestions, recommendations, best practices?

Replace syslog-ng w/ fluentd because it seems like they can both do the same things and fluentd maybe does more? Would there be a reason to keep syslog-ng and fluentd?

Could someone please help me get started by providing an example config for td-agent.conf to replace existing syslog-ng functionality by reproducing similar results as below config?

Also, is it ok / better to use elasticsearch for long term log storage for all client logs versus text file + log rotation?

If so how can I setup similar output to below config to store in elasticsearch?

Details of existing setup using syslog-ng:

On the log server, relevant excerpt from /etc/syslog-ng.conf

source s_remote_logs {
        udp(port(514));
};


destination d_remote_logs {
        file("/archive/logs/HOSTS/$HOST_FROM/$FACILITY/$YEAR$MONTH/$FACILITY$YEAR$MONTH$DAY" create_dirs(yes) );
};


log {
        source(s_remote_logs); destination(d_remote_logs);
};

And then LAN clients are setup to forward their logs from rsyslog to the log server by placing the following in /etc/rsyslog.conf

*.* @<logserver ip>:514

This setup creates a easy to browse dir/file structure on the central log server.

archive/
└── logs
    └── HOSTS
        ├── 10.10.10.10
        │   ├── authpriv
        │   │   ├── 201312
        │   │   └── 201407
        │   ├── cron
        │   │   ├── 201312
        │   │   ├── 201407
        │   │   └── 201408
        │   ├── daemon
        │   │   ├── 201312
        │   │   ├── 201407
        │   │   └── 201408
        │   ├── syslog
        │   │   ├── 201407
        │   │   └── 201408
        │   └── user
        │       ├── 201312
        │       ├── 201407
        │       └── 201408
        └── 10.10.10.55
            ├── authpriv
            │   ├── 201407
            │   └── 201408
            ├── cron
            │   ├── 201407
            │   └── 201408
            ├── daemon
            │   ├── 201407
            │   └── 201408
            ├── kern
            │   └── 201407
            ├── syslog
            │   ├── 201407
            │   └── 201408
            └── user
                └── 201407

[Masahiro Nakagawa]

Hi,

If you want to setup following flow, the configurations is simple.

ClientA----|

               |----> Fluentd ----> Elasticsearch + Kibana

ClientB----|

<source>

  type syslog

  port 5140 # default port. If you can set capability, you can set 514.

  tag syslog

</source>

<match syslog.**>

  type elasticsearch

  logstash_format true

</match>

Check this document: http://www.fluentd.org/guides/recipes/rsyslogd-aggregation

This is similar case.

If you want to store raw logs to local or object storage, you can use copy plugin.

<match syslog.**>

  type copy

  <store>

  </store>

    type file # or s3 or etc

    path /path/to/archive

  <store>

    type elasticsearch

    logstash_format true

  </store>

</match>

> is it ok / better to use elasticsearch for long term log storage for all client logs

Elasticsearch is search engine, not storage.

So I think using Elasticsearch as a storage is high cost.

> This setup creates a easy to browse dir/file structure on the central log server.

Creating same structure is difficult with standard out_file plugin.

http://docs.fluentd.org/articles/out_file#path-required

If you want same structure for file archive,

you should implement own plugin extending out_file plugin.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vmi...@gmail.com]

Thanks Masahiro,

For your quick and useful reply.

What database would you recommend for long term storage of various logs?

[Kiyoto Tamura]

>What database would you recommend for long term storage of various logs?

What kind of use case do you have in mind?

--

Check out Fluentd, the open source data collector for high-volume data streams

[vmi...@gmail.com]

As described above, I'd like to use a database for long term storage of all logs from LAN clients.

Then later use that data to send to something like elasticsearch / kibana as well,

This isn't mission critical as it's for learning purposes on my personal LAN.

[Kiyoto Tamura]

Sorry for making you repeat yourself.

Why not start using Elasticsearch right now? One relatively cheap place to store data is S3, and you might find this example useful for how to do that -> http://www.fluentd.org/guides/recipes/elasticsearch-and-s3

[vmi...@gmail.com]

Also as already mentioned previously by repeatedly / Masahiro:

Elasticsearch is search engine, not storage.
So I think using Elasticsearch as a storage is high cost

S3 is also out of the questions as I mentioned I'd like to do this all locally.

So question still stands -- 

Which database is most advisable to use to for long term storage of logs from various LAN clients?

[Kiyoto Tamura]

Hi vmixus-

>Which database is most advisable to use to for long term storage of logs from various LAN clients?

If storing is what you want, and you would rather do everything locally, local file system is a good start. If you use out_file to write distributed file systems like GlusterFS.

As Masahiro said, in order to create the file hierarchy you described in the original message, you will need to extend out_file. http://docs.fluentd.org/articles/plugin-development is a good place to start, and if you know a bit of Ruby, the source code for out_file (https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/out_file.rb) should come in handy in writing your own plugin.

I hope this helps,

Kiyoto

[vmi...@gmail.com]

Kiyoto, thanks for your input; however as stated initially, saving to text files is something that's already setup and working w/ log rotation using syslog-ng.

Writing custom plugins only to recreate what's already working doesn't seem worthwhile.

Question is meant to help determine which database (i.e. mysql, pgsql, etc.) would be most advisable to use for long term log storage as an alternative to using rotating text files?

Something that's easier to plug into elasticsearch / kibana later for gui or just run sql queries against.

Something which can perhaps compress data so as not to take up too much space and low system overhead as well.

[Kiyoto Tamura]

vmixus-

I see. If storage space is a chief concern, then I would just zip up the files. However, since you seem very keen on using a database, you might find this article helpful: https://www.digitalocean.com/community/tutorials/sqlite-vs-mysql-vs-postgresql-a-comparison-of-relational-database-management-systems

That said, unless you feel the need to analyze/access your log data in a structured way, I really recommend keeping them as (potentially distributed) log files because it is easy to move files around: putting data in and out of databases is a lot of work.

Here is a list of popular plugins to stream data into RDBMS: http://www.fluentd.org/plugins#rdbms

[vmi...@gmail.com]

Thanks kiyoto for those links, although I've seen both of those already and still neither address the original question, which I'll reiterate,

Which database would be most advisable to use for long term log storage as an alternative to using rotating text files?

I was hoping for an answer based on experience as to what DB's have worked out best for others and why they might have chosen one over the other.

[Kiyoto Tamura]

vmixus,

At the end of the day, no database solution is categorically better than others. Different databases (or more generically storage systems) have relative strengths and weaknesses, and it depends so much on your needs, requirements and constraints (for example, in your case, cloud services seem to be out of question). The best I can do is to lay out available options, and promulgating a blanket recommendation would be the worst advice I can give you.

If you can shed some more light on your requirements (beyond the fact this is for long term storage, no cloud vendor, that it needs to be a database. For example, what databases have you considered thus far?), other might be able to weigh in and share their experiences.

Kiyoto

[vmi...@gmail.com]

kiyoto,

thanx for sticking it out, I felt my use case was general enough that there would be a proven and tested recipe.

Sorry if what I'm trying to accomplish hasn't been explained clearly enough.

So far my setup gives me a organized file/dir structure (as described above) but I wanted the information in those logs to be more accessible.

The reasoning was that by sticking it all in one database it would become possible to run queries (elasticsearch) on all the data and help visualize it as well (ie. kibana)

Also, since it's all in one database instead of a infinite number of growing files it becomes easier to manage since all that needs to be done is backup the db.

Since Masahiro indicated that elasticsearch would be too high cost for a long term storage solution, I'm asking what would be a better way to store this data long term using a database?

bbb