I would like to know:- if you think that those features are needed indeed (or if some should be scrapped)
- if you think that additional features are needed for container logging
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
PROPOSITION 2: change the behavior of "docker logs", so that it doesn't send ALL the logs for a given container, because that's an unrealistic demand.- result: now that we're not asking Docker to do something impossible, we can focus on what's actually doable :-)- drawback: bad news for people who relied on "docker logs" to output ALL THE THINGS.
PROPOSITION 3: convert all messages (logs emitted by Docker itself, logs emitted by containers from stdout/stderr or syslog, events about containers...) into a common format (which serializes to JSON?)- result: we can now log "rich" messages.- drawback: requires unification of container logs and Docker logs; if this is not desirable, Docker logs can be kept out of the equation.
I'm also in favor of the "plain log files" option that you mentioned, Jérôme. As you said, this can be done regardless of which of your propositions (or some other proposition) is implemented.But it would be really useful to be able to parse out the different logs visually and programmatically. It would be nice if `docker logs` would put a marker in each line of log output, to indicate where it came from. (Even better for human scanning: colorize the markers by default, but allow colors to be turned off.)
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Alexander,
It wasn’t clear to me whether the feature creep you’re talking about it Jérôme’s proposal #4, or my addition of making it easy to parse out the different logs. From the way you inlined my comments, it looks like you were addressing that. I probably wasn’t clear in giving a reason for what I said. I apologize for that.
Here’s the reason: as a simple matter of pragmatism, if Docker were to provide a -log feature that consolidates multiple logs into the docker logs
output but were simply to commingle output from those different log files without any way to distinguish them, then the feature would be useless. A single stream of log output from multiple sources? How would that be useful? What’s the first thing people would do? They would have to write parsers to pull apart the lines based on where they came from. Those parsers would be based on heuristics, and they would have errors, and a lot of time would be wasted. There are no built-in or third-party tools that will automatically parse those log lines out from each other.
So my point is just that if proposal #4 is implemented, tagging the log lines is a must-have. Not a nice-to-have, and not feature creep. But if proposal #4 is not implemented, well, tagging isn’t needed. :-)
Brian
--
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[20:38:09] <gabrtv> good thread on fixing logs: https://groups.google.com/forum/#!topic/docker-dev/3paGTWD6xyw
[20:38:24] <crosbymichael> I've read that a few times
[20:38:28] <crosbymichael> what do you think about it?
[20:39:38] <crosbymichael> me today "i'm getting ddos'd by myself, let me check container logs to see what is going on..... I'm ddos'n myself with all logs from the entire month"
[20:41:01] <gabrtv> i don't have a strong preference for a specific solution, however
[20:41:50] <gabrtv> i tend to think docker should be doing less in the case of logs, not more
[20:41:57] <crosbymichael> me too
[20:42:10] <shykes> yeah anything that logs to disk should be a plugin
[20:42:24] <shykes> sending to syslog should be a plugin
[20:42:25] <gabrtv> i also think dumping logs into /var/lib/docker (as json!) is a bad idea
[20:42:54] <crosbymichael> gabrtv: it's dumped as json so we can split stderr, stdout
[20:43:03] <shykes> the original design was: start simple with a dump to disk (not in json). expose a network api to atomically "get and flush"
[20:43:26] <shykes> so that it's the responsibility of an outside system to collect and truncate logs via the remote api
[20:43:38] <bkc_> sounds like a good idea :)
[20:44:16] <crosbymichael> it will be a gold rush once we have plugins, so many to build
[20:44:32] <shykes> in that design the design is used as a temporary buffer
[20:44:41] <gabrtv> shykes: agree. that's essentially the same contract any other service maintains w/ system administrators.
[20:45:03] <shykes> yeah the important decision was, "let's not rely on ssh access + regular logrotate"
[20:45:12] <shykes> "let's expose it on the remote api instead"
[20:45:35] <gabrtv> shykes: so are plugins the near-term answer in your view?
[20:45:39] <shykes> what's missing currently is 1) aggregate logging (as opposed to one-shot) and 2) that actual flush-and-truncate which we never implemented
[20:45:45] <shykes> yes
[20:46:03] <shykes> well mid-term, they are really coming together
[20:46:10] <shykes> maybe a few short-term fixes before that
[20:46:14] <gabrtv> look forward to pitching in on that..
[20:46:19] <shykes> but I don't see a major overhaul of logging before plugins
[20:46:53] <shykes> unless jerome proposes an actual patch from this email thread
[20:47:36] <shykes> (or anyone else with specific suggestions in that thread)
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Hi,I'm un-burying this conversation, because I had the opportunity to talk about logging recently with various people and here are some ideas that came out of it:- Docker may inject a /dev/log socket in each container- Docker may aggregate those logging streams + stdio streams of each container- if a container logs to plain files, it can log to a volume, then that volume can be shared with another container whose sole purpose is to "tail" those files to syslog or stdout- then, this "firehose" of logs can be sent to multiple places: local file (/fifo/socket), stdin of another container, journald- Docker doesn't do any kind of buffering whatsoever; if one of the outputs of the firehose is clogged, it drops messages, and when it will unblock, it will write some information to tell that X messages were dropped (or at least, that some messages were dropped)By default, the firehose could be sent to journald (when available), local file (with logrotate rules setup by distro scripts), stdout (otherwise).
- Docker may aggregate those logging streams + stdio streams of each container
- if a container logs to plain files, it can log to a volume, then that volume can be shared with another container whose sole purpose is to "tail" those files to syslog or stdout
- Docker doesn't do any kind of buffering whatsoever; if one of the outputs of the firehose is clogged, it drops messages, and when it will unblock, it will write some information to tell that X messages were dropped (or at least, that some messages were dropped)
How does that sound?
Hi Jerome,On Wed, Feb 26, 2014 at 1:29 AM, Jérôme Petazzoni <jerome.p...@docker.com> wrote:- Docker may aggregate those logging streams + stdio streams of each containerImo it should be possible to retain the source of a log line. Not sure how the aggregation would look like though.
- if a container logs to plain files, it can log to a volume, then that volume can be shared with another container whose sole purpose is to "tail" those files to syslog or stdoutBut that's nothing Docker needs to support, right? IMO Docker should provide only one way to log but make it possible to use a "filter" (think svlogd) which converts the input to whatever Docker supports.
- Docker doesn't do any kind of buffering whatsoever; if one of the outputs of the firehose is clogged, it drops messages, and when it will unblock, it will write some information to tell that X messages were dropped (or at least, that some messages were dropped)I'm not so sure about that. In general I agree, we can't block everything if the firehose is clogged, but there are scenarios where loosing a log line can be an severe issue. Think of security/auditing logs or statistics that will be generated from the logs. I think what you proposed is a good default, but we might want to make this configurable.
TRUSTED JOURNAL FIELDS Fields prefixed with an underscore are trusted fields, i.e. fields that are implicitly added by the journal and cannot be altered by client code. _PID=, _UID=, _GID= The process, user and group ID of the process the journal entry originates from formatted as decimal string.
...
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
It *exactly* this kind of feature creep I'm worried about. Docker is a *piece* of the linux/unix echosystem, not a replacement for it. All the other logging systems have features like this, plus a boatload more, including things like remote logging, rate limiting, log rotation, log indexing, secure metadata for log rows, etc.
We can either make sure we integrate well with these systems, or we'd be on an eternal mission to replicate all those features (that were added because people need them). And, in practice I think everyone who runs docker in production already have an existing production logging setup, so they won't use the docker log command anyway.
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi everyone, thanks for sharing your (very diverse!) opinions on this thread. I thought it would be a good time for an official maintainer to weigh in. Just a few quick points:
Hey list,
This is an attempt to gather ideas about improving the handling of container logs.This is not about logs of Docker itself (see https://github.com/dotcloud/docker/issues/936 for that); we're talking about the logs generated by the containers here.Current situation:- nothing special is done regarding syslog (if a process tries to use standard syslog calls, that will go to /dev/null since /dev/log won't exist, unless you run syslog in the container).
- nothing special is done regarding regular log files (e.g. if you run something that writes log to /var/log/blah/blah.log, it will just stay here).- stdout+stderr of processes running in containers are captured by Docker, and stored in a JSON format looking like this:{"log":"Creating config file /etc/mercurial/hgrc.d/hgext.rc with new version\n","stream":"stderr","time":"2013-11-01T13:51:19.763621802-07:00"}- those log files are stored to disk, and grow boundlessly- logs can be consumed entirely (with "docker logs") or kind of streamed (with "docker attach"), but it's not possible to stream from a given point, or consume only parts of the logsIdeally, we want to capture more log sources (syslog and regular files come to mind), and better ways to consume logs.Specifically, it would be nice if we could...1) handle log entries sent to syslog, since many unix daemons use that, and it allows to carry some extra info (facility and priority)2) handle regular logfiles, since some programs will use that (and sometimes different log files will have different meanings, e.g. access.log and error.log)3) store log entries in a bounded ring buffer, to make sure that logs will never fill up disk space by default4) stream log entries, and be able to resume the stream (if the stream breaks) without losing entriesI would like to know:- if you think that those features are needed indeed (or if some should be scrapped)- if you think that additional features are needed for container logging- if you are willing to work on implementing that kind of stuffI have some ideas about how to implement this, but before, I'd like to get feedback on the general idea.Thank you!
--@jpetazzoLatest blog post: http://jpetazzo.github.io/2013/11/17/flynn-docker-paas/
--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the Google Groups "docker-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/docker-dev/3paGTWD6xyw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to docker-dev+...@googlegroups.com.