Using lnav with jsonevent layout.

[gol...@i-teco.ru]

Hi, I am very attracted by lnav features and try to use it with logs in jsonevent layout [1]. Example of such log (named trace.json) is here [2]. My config is [3].

My goal is to filter log by complex criteria like "level >= 'INFO' and logger_name = '...' and jget(mdc, '/url') = '...' or ...". Is it possible now and how?

I thought this may achieved via sql functionality and tried folowing.

I open log file trace.json, hit ":goto 235", so log row at the top has timestamp 2015-11-16T14:13:35.519 and message "Connecting". When I hit ";" (starting to write sql query), I see "Known message fields: (SQL table -- opcua_log): ..." and list including "logger_name" and "mdc", and also "JSON fields: ... jget(mdc, '/url')".

But I cannot search by these fields: ";select logger_name from opcua_log" outputs "error: no such column: logger_name", so does ";select * from opcua_log where logger_name like 'ru.%'".

May be this is a bug or I do wrong thing.

I am testing lnav 0.7.3 on Centos 7, x86_64; lnav 0.8.0 have similar behaviour.

PS. Another interesting questions are:
* How to hide field "@version"? It is not informative.
* Some rows have pair, which may be identified by correlation id in mdc field (like jget(mdc, '/uuid') and jget(mdc, '/future') in my example log). How can I jump to paired row?

[1] https://github.com/logstash/log4j-jsonevent-layout
[2] https://drive.google.com/file/d/0B9tzMFKEeClfMFNfbUpzMmdlRkU/view?usp=sharing
[3] http://pastebin.com/9u8PJREM and also below:
{
"opcua_log" : {
"title" : "OpcUa-client log",
"description" : "logstash-json-layout",
"json" : true,
"file-pattern" : "trace\\.json.*",
"level-field" : "level",
"body-field" : "message",
"timestamp-field": "@timestamp",
"line-format" : [
{ "field" : "@timestamp", "default-value": "∅" },
" [", { "field" : "logger_name" },
"]\t ", { "field" : "message" }
]
}
}

[Timothy Stack]

On Wed, Nov 18, 2015 at 6:27 AM, <gol...@i-teco.ru> wrote:

Hi, I am very attracted by lnav features and try to use it with logs in jsonevent layout [1]. Example of such log (named trace.json) is here [2]. My config is [3].

​Thanks​ for providing your config and log, it makes things easier.

 

My goal is to filter log by complex criteria like "level >= 'INFO'

​You can ​use the 'set-min-log-level' command to set the minimum level when displaying messages,

for example:

​  :set-min-log-level info​

​This will also affect which ​messages are included in sql results.

and logger_name = '...' and jget(mdc, '/url') = '...' or ...". Is it possible now and how?

I thought this may achieved via sql functionality and tried folowing.

​I think ​you should be able to do what you want with the SQL interface.

 

I open log file trace.json, hit ":goto 235", so log row at the top has timestamp 2015-11-16T14:13:35.519 and message "Connecting". When I hit ";" (starting to write sql query), I see "Known message fields:  (SQL table -- opcua_log): ..." and list including "logger_name" and "mdc", and also "JSON fields: ... jget(mdc, '/url')".

But I cannot search by these fields: ";select logger_name from opcua_log" outputs "error: no such column: logger_name", so does ";select * from opcua_log where logger_name like 'ru.%'".

​Even for json logs, the log format still needs to define the columns that can show up (i.e. the

"values" field in the format).  I think that is the only thing missing from your format.  I've

attached an updated format file with the definitions and that seems to work for me.

 

May be this is a bug or I do wrong thing.

​​More just bad/incomplete documentation.

 

I am testing lnav 0.7.3 on Centos 7, x86_64; lnav 0.8.0 have similar behaviour.

PS. Another interesting questions are:
* How to hide field "@version"? It is not

​​

informative.

​Unfortunately,​ I don't think there is a way to do this at the moment.  There is a 'hide-extra'

flag that will hide all extra fields and only show what is used in the "line-format", but there

isn't a per-field option to hide.  I'll add a github issue for that.

 

* Some rows have pair, which may be identified by correlation id in mdc field (like jget(mdc, '/uuid') and jget(mdc, '/future') in my example log). How can I jump to paired row?

​This i​s a feature I've been meaning to add for awhile and haven't gotten around to.  For

now, I think you'll have to settle for regular string searches.

​thanks,

tim​

 

[1] https://github.com/logstash/log4j-jsonevent-layout
[2] https://drive.google.com/file/d/0B9tzMFKEeClfMFNfbUpzMmdlRkU/view?usp=sharing
[3] http://pastebin.com/9u8PJREM and also below:
{
    "opcua_log" : {
        "title" : "OpcUa-client log",
        "description" : "logstash-json-layout",
        "json" : true,
        "file-pattern" : "trace\\.json.*",
        "level-field" : "level",
        "body-field" : "message",
        "timestamp-field": "@timestamp",
        "line-format" : [
            { "field" : "@timestamp", "default-value": "∅" },
            " [", { "field" : "logger_name" },
            "]\t ", { "field" : "message" }
        ]
    }
}

--
You received this message because you are subscribed to the Google Groups "lnav" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lnav+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Головко Алексей]

Thank you very much!

Your config much more usable.

"hide-extra" flag is enough.

Now i am planning to play more with lnav some time to learn it.

---

From: Timothy Stack <timothys...@gmail.com>
Sent: Wednesday, November 18, 2015 8:21 PM
To: Головко Алексей
Cc: lnav
Subject: Re: Using lnav with jsonevent layout.