Color by line rather than field?

[Paul Long]

As a newbie, I think I understand that lnav has identified some fixed fields and types that it will colorize.  However, a workflow I prefer is to colorize lines based certain events.  For instance, a line is colored Green when a service starts and Red when it stops.  I'm guessing this functionality doesn't exist?  If so, could it make sense to add a feature to colorize a line?

My thinking is that it could exist as another window, similar to the Text Filters.  Or maybe just a extra field to IN filters that indicates a color.  One challenge could be ordering might become important, as you might want to prioritize, and therefor need a way to reorder or specify priority.  It could be used in combination with highlights and there probably needs to be a way to disable the current color filtering.

In case it's useful, here's a scenario that covers my usage.  Some of this is already possible I believe.  

I open up a set of logs covering the last week of logs and apply a scheme for troubleshooting a specific problem.  My goal is to view a log and see if I can find a problem we've discovered in the past.  This scheme/view can be shared with other users.  When I apply the scheme, it filters out most of the noise and shows colorizes important events like starts/stops, performance based events, and interesting status messages.  The log that was a 100,000 lines is condensed to 2000.  I can easily enable an IN filter for various components using a keystroke, ctrl+u for our UX component, for instance.  Next I want to follow the logging of a GUID which represents a transaction in our components, so that field is highlighted without losing the background color.

Is this something you think the lnav architecture would handle?  Do you think it would straight forward to implement this if I were to try and fork the git project?  Are there any plans like this or other related projects already?

Thanks, Paul

[lnav support]

I think you're looking for the ":highlight" command:

   https://docs.lnav.org/en/latest/commands.html#highlight-pattern

To get finer control over the display, you can also add highlights

to a log format, see:

   https://docs.lnav.org/en/latest/formats.html

Note that you can patch an existing format, like syslog_log, you

don't necessarily have to create a new one.  See:

   https://docs.lnav.org/en/latest/formats.html#patching-an-existing-format

Tim

------- Original Message -------

--
You received this message because you are subscribed to the Google Groups "lnav" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lnav+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lnav/c5aa50dc-47db-4c69-9cab-8edfa8c42594n%40googlegroups.com.

[Paul Long]

Can I associate specific colors using the Highlight command so that one highlight is green and another is red?  I guess it's not the whole line, which might be ok, but then can I remove the default coloring that occurs so that does not add noise I don't want?

And I think you are saying I can add these highlights automatically to a format file?

Thanks, Paul

[lnav support]

On Tuesday, August 23rd, 2022 at 1:07 PM, Paul Long <paule...@gmail.com> wrote:

Can I associate specific colors using the Highlight command so that one highlight is

green and another is red? 

Not at the moment, no.  Maybe in the future since I've made some recent changes that

would make it easier to add flags to commands.

I guess it's not the whole line, which might be ok, but then can I remove the default coloring that occurs so that does not add noise I don't want?

What default coloring are you speaking of?  Can you send a screenshot?

You can tweak the theme definitions to your liking using the ":config" command, see:

    https://docs.lnav.org/en/latest/config.html#theme-definitions

Or, just switch to the grayscale theme -- ":config /ui/theme grayscale" -- Most of the

coloring is turned off in that theme.

And I think you are saying I can add these highlights automatically to a format file?

I'm attaching a file named "syslog_highlighter.json" that you can put in your

"~/.lnav/formats/installed" directory (or ~/.config/lnav/...) that adds a highlighter

to syslog files.  Here's a screenshot of what it looks like for me:

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/lnav/42e12826-e63d-4b85-bea2-e2ef2a985a0cn%40googlegroups.com.

[Paul Long]

Here's a screenshot, but I had to obscure it as I'm not sure I can share these externally.  But I think you can get the idea.  And based on what you are saying above, I think I can create a simple theme to get rid all of it if I want.  Then I can place highlight with colors in my format, which I've created already, so maybe I can solve the problem that way.  They won't be dynamic, but maybe I can solve that with IN/OUT filters.  So I will give that all a try.

The last missing part is being able to apply different filter sets for different problems.  I thought I saw somewhere in the documentation that I can run a script that can execute a file that consists of commands?  Or maybe create different sessions with different filters in them that I can bring in using load-session?  Ideas on if there is a good way to do that?

Thanks, Paul

[lnav support]

Actually, let's take a step back... lnav has more flexibility than just highlighting.

There is some tagging and commenting functionality that might be easier to

understand than just plain highlighting.  These features allow you to annotate

log messages and those annotations will be visible underneath the messages,

see:

  https://lnav.org/2018/05/17/tags-and-comments.html

There's some more explanation in the "Taking Notes" section of the docs:

  https://docs.lnav.org/en/latest/usage.html#taking-notes

I've also recently added the ":export-session-to" command for exporting the

current session state, see:

  https://docs.lnav.org/en/latest/usage.html#sharing-sessions-with-others

As for automating filtering, you can write lnav scripts, see:

  https://docs.lnav.org/en/latest/formats.html#scripts

You can also access the filtering state through the SQL tables as described

here:

  https://docs.lnav.org/en/latest/sqltab.html#lnav-view-filters

Tim

------- Original Message -------

To view this discussion on the web visit https://groups.google.com/d/msgid/lnav/d82f26e6-37ac-44a0-ba4a-611d8c847846n%40googlegroups.com.

[Paul Long]

Thanks, I'll look into those in more details.  I looked at the comment and notes stuff, which is useful, but I want it to be automated and something I can apply to any set of logs.  The scripts look promising, so maybe I can leverage that.

I did try to implement a blank template, but I couldn't get it to work.  The documentation wasn't clear to me, but what I did was copy the default.json.sample file to the installed directory.  I named it blank.json, then started lnav and attempted to load it wtih :/config /ui/theme blank, and it says error: command: 1:unknown theme -- blank.  What am I doing wrong?

Paul

[lnav support]

On Tuesday, August 23rd, 2022 at 5:57 PM, Paul Long <paule...@gmail.com> wrote:

Thanks, I'll look into those in more details.  I looked at the comment and notes stuff, which is useful, but I want it to be automated and something I can apply to any set of logs.

Adding comments and tags can be automated through SQL.  This is covered in the

"Taking Notes" section I mentioned.

The scripts look promising, so maybe I can leverage that.

I did try to implement a blank template, but I couldn't get it to work.  The documentation wasn't clear to me, but what I did was copy the default.json.sample file to the installed directory.

You copied the "default-theme.json.sample" file to "~/.lnav/configs/installed" directory

with the name "blank.json"?  I'm not quite clear on the steps you took.

  I named it blank.json, then started lnav and attempted to load it wtih :/config /ui/theme blank, and it says error: command: 1:unknown theme -- blank.  What am I doing wrong?

Start lnav with "-d /tmp/lnav.dbg" and then go through that debug log to see if it's

getting loaded.

Paul

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/lnav/fb0fda77-45ed-4aaa-ab57-6c22545e3f32n%40googlegroups.com.

[Paul Long]

Does automating the tags through SQL require that parts of the logs I need tagged are parsed out using a format file?  Or can the auto field detection (field=value) be used?  Right now I haven't figured out how to full parse our logs, though I still intend to try.  Just worried about performance.  As I mentioned before our logs aren't formatted very consistently so it's not easy creating regex that can identify each line separately.  

To test the theme creation I copied default-theme.json.sample to ~/.lnav/configs/installed/my.json.  This time I didn't change the file at all.  Here is a listing.

✘ paul@AZSDL-WL5CG9036P64  ~/.lnav/configs/installed  ls
blank.json blank.lnav grayscale.json.sample log.txt my.json

To try and load my theme, I started lnav with the -d param and waited ~10s so I would see a gap in the logs.  Then I waited ~10s more before I closed lnav.  This section of the log below is what I see, but I don't see that it's telling me anything interesting.

2022-08-24T09:37:24.141 E t0 session_data.cc:418 unable to upgrade bookmark table -- duplicate column name: tags │
│2022-08-24T09:37:42.634 D t0 hotkeys.cc:155 executing key sequence x3a: :prompt command │
│2022-08-24T09:37:42.634 I t0 command_executor.cc:110 Executing: prompt command │
│2022-08-24T09:37:44.927 I t0 command_executor.cc:110 Executing: config / │
│2022-08-24T09:37:47.066 I t0 command_executor.cc:110 Executing: config /th │
│2022-08-24T09:37:49.290 I t0 command_executor.cc:110 Executing: config /ui/theme │
│2022-08-24T09:37:51.096 I t0 command_executor.cc:110 Executing: config /ui/theme my │
│2022-08-24T09:37:53.695 I t0 command_executor.cc:110 Executing: config /ui/theme my │

│2022-08-24T09:38:03.926 D t0 hotkeys.cc:155 executing key sequence x71: |lnav-pop-view ${keyseq}    

The version I'm using is: Version: lnav 0.10.1-dirty

Paul

[Paul Long]

If anybody has an idea how to create and use a theme?

Paul

[lnav support]

You need to change the JSON file.  The name of the theme is in the file

itself, it's not derived from the filename.  The object to change is in the

"theme-defs" object.  For example, the monocai theme name is here:

  https://github.com/tstack/lnav/blob/fd956c66d3663a792baf194d8c211b54eb865bc4/src/themes/monocai.json#L5

If you copied the default theme, you'll need to change "default" to "blank".

On Wednesday, August 24th, 2022 at 6:45 AM, Paul Long <paule...@gmail.com> wrote:

Does automating the tags through SQL require that parts of the logs I need tagged are parsed out using a format file? 

The first example in the docs I mentioned shows exactly that, it's just

matching on the body of the log message.

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/lnav/e5230afd-4a8e-467d-b9ba-cdd76a458e93n%40googlegroups.com.

[Paul Long]

Hurray, that worked!  Guess since it was named default in the json file I didn't pick it out. Should have tried a different one :)

I'll go take all the info above and report back.  

Thanks!

Paul