Lot's of good ideas in this thread. The trick when looking at
interesting tech is figuring out how it can be applied in a way that
helps solve your problem. In my opinion the primary observability
problem Kiali tries to solve it to surface issues in the mesh, and
then helping the user get started with an investigation of the
problem. With respect to logs I think there are three aspects for
Kiali to consider:
1) Logs as a health indicator
This is not deep analysis of logs, anything like that would be
deferred to another tool. That tool that may itself generate
alerts/events to incorporate into Kiali in a more generic fashion (a
hook we don't currently have). But some level of cursory background
log crawling could be useful. Caina already has server-side ability
to turn logs into simple entries like {timestamp, level, message}
which could be useful today for log-level hits, or going forward
possibly allowing user-configured patterns (maybe from the health
config). (note, as he mentioned, complicated parsing is not what we
do, nor is it in scope.)
2) Log visualization
I think we're doing a pretty good job of this already. Mike has
added a nice split-view comparison with strong client-side filtering
ability, as well as full-window viewing. From the tech Mike showed,
and Joel commented on, an integrated view could be useful,
especially in...
3) Logs as part of a correlated view
We have some work to do here, but Caina is currently adding
server-side support for time-periods, allowing us to focus on an
interesting past window that could be determined via charts, or
graph replay, etc. I could imagine then some correlated view
involving logs, this time maybe with an integrated view of multiple
containers' logs.
The main point of this comment is to reinforce what Lucas was
saying, keep in mind the problem we're solving, and use it to limit
scope. We are likely not going to be an alerting system, nor a log
analysis tool, nor a machine learning tool. But we may be able to
leverage those sorts of things, or scratch the surface on our own
when it helps Kiali users.