Mapping logs -> programs?

[Joshua Barratt]

I've dug through the docs a few times and I still feel like I'm missing something.

Is there a way to map logs -> programs?

For example, I've configure mtail to process apache & nginx logs. I have 'apache.mtail' and 'nginx.mtail'. 

It seems like a lot of waste to run the nginx program on all the lines of the apache log, and vice versa.

Thanks for any guidance.

Josh

[Jamie Wilkinson]

The previous (internal to Google) version did do that, with a clause in the program itself naming the files that it would accept.  It's not great, because your log names may be different!  I can't yet think of a good way to configure the mapping, so right now all log lines go to all VMs.  It was cleaner to do that with the new version than propagate the feature.

I'm keen to hear your ideas on how you'd like the config interface to work, though, I agree it's wasteful if you know a priori what lines the VM will do work on.

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/faa10a14-f131-40ff-8c99-131fc4d589aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joshua Barratt]

Great, thanks for getting back to me! I see the conflict with not wanting to change the config lines for sure.

How about naming the programs on the command line?

Right now I have:

/usr/bin/mtail --progs /etc/mtail --logs /var/log/apache2/virtual_access_log,/var/log/nginx/access.log,/var/log/nginx/error.log

and

/etc/mtail/nginx.mtail

/etc/mtail/apache.mtail

What about allowing multiple --logs arguments, with optional (program=) prefixes?

/usr/bin/mtail --progs /etc/mtail --logs apache=/var/log/apache2/virtual_access_log --logs nginx=/var/log/nginx/access.log,/var/log/nginx/error.log

So logs are free to change, but you only need to make changes at the mtail invocation level, which you'd need to do anyway if the logs changed.

Seem plausible? I'm a complete golang beginner, but if you don't mind getting extra constructive/critical on a code review, I'm ok to try a PR.

Josh

[Jamie Wilkinson]

I was thinking something similar, an additional commandline flag with comma separated program:logfile pairs.  I think your suggestion is better than what I was thinking, because there's no repetition of configuration.  The concern I have is backwards compatibility; if we can retain the old meaning of the logs flag and also support the new one that would be perfect.

I can give you an extra constructive review on the PR, for sure. ;-)

The important part will be changing the way the log lines get from the log tailer to the VM loader.  Right now it is a single channel of log lines.  Instead of a single channel, there would have to be a mapping of VMs to channels that the tailer would use to send lines from each log to the right channel, or that the log file name would be passed along with it into the loader, who would then do the fanout.  I'm not sure which approach I like though, both feel like they're exposing too much information on either side of the interface which will be required to set up the mappings.  I think I slightly prefer the channel per VM approach though.  The loader and tailer both don't initialise the lines channel, they get it from the parent mtail object.  But instead, the tailer could have a pointer to the loader, and ask it for the channel to the VM it's trying to send to with a new method on the loader.

Let me know if you get stuck on the implementation, and thanks for the offer!  It'd be excellent to have this feature.

To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/b3e3cc8a-de30-4f31-b0ad-f83a7f91b853%40googlegroups.com.

[Matt Bostock]

On Tuesday, 1 September 2015 02:19:50 UTC+1, Jamie Wilkinson wrote:

The previous (internal to Google) version did do that, with a clause in the program itself naming the files that it would accept.  It's not great, because your log names may be different! 

I really like that model - it's very simple and it's trivial for someone to fork an example program and adapt it for their own configuration.

[Jamie Wilkinson]

Thanks, that is a good bit of feedback.

Josh's suggestion of another flag mapping logs to programs would be fine if there are a few logs and programs you wish to map, but I think maintenance of that would get really messy once you get to about 5 or more programs, but in that case you would really want to have the log->program isolation.

One reason I haven't tried to implement this is because I want to try to offer a standard library of mtail programs for common applications, to encourage reuse, and I haven't thought of how to do that in a platform independent way -- every distribution and OS keeps their logs in different filename paths ;-)

Brainstorming: make the log->program flag a repeatable one, so you could say:

mtail --logprog=apache:/var/log/apache/error.log --logprog=ntp:/var/log/ntp/peerstats --logprog=ntp:/var/log/syslog

which means you are not trying to write all that without any whitespace and too much punctuation (e.g. I can think of doing:

mtail --logprog=apache:/var/log/apache/error.log,ntp:/var/log/ntp/peerstats,...

which seems like a crap interface for us to manage.

Issue filed: https://github.com/google/mtail/issues/35

--

You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/b29e0056-7e2e-4040-8fed-0a76a34c378d%40googlegroups.com.