Logging stuff

[Mike Thompson]

In looking at other logging analyzers I found this one to be interesting: 

http://lnav.org/features

Wondering if we shouldn't incorporate something like their timeline or pretty print features?

-- Mike

[Joel Takvorian]

Looks like it is a compiled executable, so not sure how we could incorporate it ...

But otherwise +1 to look for existing libraries!

--
You received this message because you are subscribed to the Google Groups "kiali-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kiali-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CACms5tuj2hcPWi-mn3Q7UeOK9cfTT4Qg7FtSc27WRd3uyGDqRA%40mail.gmail.com.

[Lucas Ponce]

We should put a feature context on this.

What do we want to accomplish by adding this library ?

Kiali won't be a generic tool for logging analytics, it will correlate logs always in the context of ServiceMesh/Istio scenarios.

I'd like to see more feature proposals (i.e. log structure for a container is open, but we know the structure of the proxy, that would be fixed and probably there are possibilities having that info).

So, I'd try to identify a technical framework / library in the context of a feature we want to work on.

Lucas

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CAJo5TFmjghnsC67NPNd1ZULJ0a21jqrUkWeAOBS8EUoiL19O4w%40mail.gmail.com.

[Mike Thompson]

To further clarify, the idea wasn't to pull in the compiled code but to replicate some of the features in our logging tab via React. For example:

1) Pretty Print View:

2) Timeline View:

[Joel Takvorian]

What I liked from what I've seen in this tool is the ability to merge several log sources into one output (with some different colors).

That could be pretty handy to see istio-proxy and main container logs at the same time, instead of a split window

[Joel Takvorian]

(PS : i didn't check the latest developments in the area, perhaps that's something already in the pipeline!)

[Edgar Hernández]

I only gave it a quick look at the homepage and it looks quite awesome.

Having said that, I don't think that "replicating the features via React" is that easy.
It requires parsing... and even a so-so decent parser requires quite a lot of work.

I would delegate that work to other tools, similarly to how we are delegating some things to Jaeger and Grafana.
Perhaps, for logging it would mean to delegate to elasticsearch?

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CAJo5TF%3DQO4C5FoTx452eij1GL79UJUX5OcHpuMVfUyOd9MWyOA%40mail.gmail.com.

[Mike Thompson]

It requires parsing... and even a so-so decent parser requires quite a lot of work.

Actually, Caina has started parsing on the server-side. So we will have more options there. 

Perhaps, for logging it would mean to delegate to elasticsearch?

We could delegate to something like Logstash/Loki but I think our purpose here is just to show whatever the k8s API gives us in more powerful and easier ways. Also to correlate other information such as Health/spans/etc... with the logs.

[Joel Takvorian]

Looking around at some of the JS ecosystem, I'm close to think we could start creating our own log analyzer. Not a big thing, just something able to scan, highlight, compute some limited stats, all of that at a small scale, ie. not from hundreds of sources like ES or Loki would do, but just limited to the scope of a couple of sources (like what we currently have: workload logs, which is N pods x M containers).

Not saying ES or Loki are uninterested, of course not, but it's another degree of work with many implications.

BTW just found that: react-lazylog ( https://github.com/mozilla-frontend-infra/react-lazylog ) , from mozilla. It's purely visualization, no analysis afaict. Might be interesting.

[Caina Costa]

On Tue, Oct 13, 2020 at 1:15 PM Joel Takvorian <jtak...@redhat.com> wrote:

Looking around at some of the JS ecosystem, I'm close to think we could start creating our own log analyzer. Not a big thing, just something able to scan, highlight, compute some limited stats, all of that at a small scale, ie. not from hundreds of sources like ES or Loki would do, but just limited to the scope of a couple of sources (like what we currently have: workload logs, which is N pods x M containers).

+1.

I think we can give a more holistic approach that ES/loki can, because of the other data we already get.

 

Not saying ES or Loki are uninterested, of course not, but it's another degree of work with many implications.

BTW just found that: react-lazylog ( https://github.com/mozilla-frontend-infra/react-lazylog ) , from mozilla. It's purely visualization, no analysis afaict. Might be interesting.

On Tue, Oct 13, 2020 at 4:12 PM Edgar Hernández <eher...@redhat.com> wrote:

I only gave it a quick look at the homepage and it looks quite awesome.

Having said that, I don't think that "replicating the features via React" is that easy.
It requires parsing... and even a so-so decent parser requires quite a lot of work.

Seems ok to me. Long term projects are ok, and I think we can do better than others because we can do less instead of more. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CAJo5TF%3DpPK6xrrAAmsPxAWksCUPuNoc0sZLZHMsn90JhSQOONA%40mail.gmail.com.

[Caina Costa]

On Tue, Oct 13, 2020 at 1:11 PM Mike Thompson <mith...@redhat.com> wrote:

It requires parsing... and even a so-so decent parser requires quite a lot of work.

Actually, Caina has started parsing on the server-side. So we will have more options there. 

Yes, but I would say at this moment "parsing" is a strong word. We get some data from the text, but do not generate a structure. This is the case because log format changes a lot depending on the runtime you're on.

There is some consistency, though, namely on the fact that timestamps are stable because k8s is the one adding them.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CACms5tuOfrUj5Wd94%2Bj%2BjUZ_QHowwyb6Dezj--QiRDOew8Xcbw%40mail.gmail.com.

[Jay Shaughnessy]

Lot's of good ideas in this thread.  The trick when looking at interesting tech is figuring out how it can be applied in a way that helps solve your problem.  In my opinion the primary observability problem Kiali tries to solve it to surface issues in the mesh, and then helping the user get started with an investigation of the problem.   With respect to logs I think there are three aspects for Kiali to consider:

1) Logs as a health indicator

This is not deep analysis of logs, anything like that would be deferred to another tool.  That tool that may itself generate alerts/events to incorporate into Kiali in a more generic fashion (a hook we don't currently have).  But some level of cursory background log crawling could be useful.  Caina already has server-side ability to turn logs into simple entries like {timestamp, level, message} which could be useful today for log-level hits, or going forward possibly allowing user-configured patterns (maybe from the health config).  (note, as he mentioned, complicated parsing is not what we do, nor is it in scope.)

2) Log visualization

I think we're doing a pretty good job of this already.  Mike has added a nice split-view comparison with strong client-side filtering ability, as well as full-window viewing.  From the tech Mike showed, and Joel commented on, an integrated view could be useful, especially in...

3) Logs as part of a correlated view

We have some work to do here, but Caina is currently adding server-side support for time-periods, allowing us to focus on an interesting past window that could be determined via charts, or graph replay, etc.  I could imagine then some correlated view involving logs, this time maybe with an integrated view of multiple containers' logs.

The main point of this comment is to reinforce what Lucas was saying, keep in mind the problem we're solving, and use it to limit scope.  We are likely not going to be an alerting system, nor a log analysis tool, nor a machine learning tool.  But we may be able to leverage those sorts of things, or scratch the surface on our own when it helps Kiali users.

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CAJvPVZ6LyATYhpuG_U7Hdtw-jX6tooy0x7%2BpkW-M9PysdgjMag%40mail.gmail.com.

[Mike Thompson]

Yes, that React-lazylog does look interesting! I had not seen that. Perhaps we can use it instead of text boxes?

[Lucas Ponce]

Also, I'd add to think in our domain.

Generic logging (or App logging) it's a huge domain, but in our domain / problem we have some advantage:

- We already know in advance the structure of the proxy logs.

- Same for the log of core components of Istio (i.e. istiod / ingress / egress pods).

To me, Kiali would be the ideal tool to extract, interpret and correlate that specific info with all signals and information we are collecting.

(The split view of proxy + container logs is great; it can show what the sidecar is doing).

To also add more feature context, read point n 4 of this article in case it may help of inspiration:

https://developer.ibm.com/articles/4-steps-to-debug-your-edge-microservices-in-istio-service-mesh/

Let's think about what Kiali can automate and it's part of the ServiceMesh scenario we are helping to simplify.

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/40577160-885d-aea3-e764-d8413933d193%40redhat.com.

[Mike Thompson]

Lots of really good conversation here. Perhaps we can circle back on this in our Water Cooler (agenda Item)?

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CAGUuq3UtpH0pWZxLfuPaym0iDw3K54QYeaEd9FS%3DNPi5tSKuYw%40mail.gmail.com.

[Alissa Bonas]

Are there any actions items/future feature plans that came up from this discussion? (yes, I know, I'm late to this thread :))

To view this discussion on the web visit https://groups.google.com/d/msgid/kiali-dev/CACms5tuYcnPg8GojghC%3DTiJwUrE00bNbA6X4%2BHepCRm_8cUa2g%40mail.gmail.com.