lnav v0.14.0 released!

[lnav support]

Hi Folks!

lnav v0.14.0 has been released with quite a few

updates:

  https://github.com/tstack/lnav/releases/tag/v0.14.0

Here's the NEWS:

Features:

* The Filter configuration panel in the TUI now supports

  editing the minimum log level and minimum/maximum

  times for the current view.  Previously, only the

  `:set-min-log-level`, `:hide-lines-before`, and

  `:hide-lines-after` commands could be used to set the

  values.  Pressing `l` will create/set the log level.

  Pressing `m` will create/set the minimum

  time and pressing `Shift` + `M` will create/set the

  maximum time.  Live preview has also been added to

  show which lines will be filtered out when the min/max

  is applied.

* The `-S`/`--since` and `-U`/`--until` flags have been

  added to limit how much of a log file is indexed.

  Supported values are relative (e.g. "yesterday",

  "30 min ago") or absolute local times ("2020-01-01").

  The values on the main command-line set the defaults

  for all values that are opened.  The `:open` command

  supports the same options to change the values from

  the default for a particular file.  Files with

  content that lie completely outside of the cutoff

  will be closed to reduce resource usage.

* Broadened support for viewing files on remote hosts.

  The "tailer" program that is transferred to the

  remote host to monitor files and perform other tasks

  has been translated to Python3.  If the remote host

  does not have Python, lnav will fall back to the APE

  binary.

* Highlights can now be applied to a particular field

  instead of the whole line.  For log formats, a

  `value` entry can now have a `highlights` object

  that contains definitions for the pattern to match

  and the style to apply.  For interactive use, the

  `:highlight-field` command was added.

* The "Files" panel now shows a progress bar for each

  file as it is being indexed and finishes with a

  check-mark if indexing was successful, a warning

  sign if the file has some notes, or an error mark

  if something else happened.

* The `:write-jsonlines-to` command now supports the LOG

  view.  When lines are marked in the LOG view, the command

  will write each marked log message as a JSON object with

  the standard log fields (`log_path`, `log_time`,

  `log_level`, `log_opid`, `log_line_link`) along with any

  format-specific parsed values, comments, and tags.

  The `--all` flag can be used to write all visible log

  lines instead of only marked lines.

* Added a log format for MongoDB's structured JSON log

  format (4.4+).

* Added a log format for the Robot Framework's debug

  log file format.

* Added `"timestamp"` as a value kind for log format

  definitions.  Fields with this kind will be parsed

  and displayed in the local timezone, and SQL

  queries will return them in a normalized format.

* Added the `:toggle-sticky-header` command to pin a

  line to the top of the view as a sticky header.

  Sticky headers remain visible as you scroll past

  them, making it easy to keep context lines in view.

  The `:clear-all-sticky-headers` command removes all

  sticky headers in the current view.  A hidden

  `log_sticky_mark` column is available in log tables

  to get/set the sticky state via SQL.  Sticky headers

  and user bookmarks are saved and restored across

  sessions for both log and text views.  The command

  is bound to `CTRL+S` by default.

* In pager mode (activated by the `-q` flag), the input

  will always be written to the terminal if it is

  smaller than the height.  Previously, if the input

  took awhile to generate, nothing would be written.

  Also, when the input is larger than the terminal

  height, any marked lines will be written to the

  terminal.

* Introducing "Log-Oriented Debugging", a collection of

  features to streamline mapping log messages back to

  the source code that generated them.  For example,

  given the log message "Hello, Steve!" and the source

  directory containing the log statement.  lnav can

  find the line of code that generated the message,

  such as `logging.info("Hello, %s!", name)`, and

  determine the value of the substituted variables

  (`name` => `Steve`). This functionality is

  implemented using the

  [log2src](https://github.com/ttiimm/log2src) project.

  The following features have been added in support of

  this functionality:

  - The `:add-source-path` command was added to specify

    the source directories to be scanned for log

    statements.

  - Log formats can now specify source file/line and

    thread ID with the `src-file-field`, `src-line-field`,

    `src-location-field`, and `thread-id-field` properties.

    These fields can then be accessed in the SQL vtables

    as `log_src_file`, `log_src_line`, and `log_thread_id`.

  - The `:breakpoint`, `:toggle-breakpoint`, and

    `:clear-breakpoints` commands have been added to

    support setting/clearing breakpoints for log messages.

    The `CTRL-B` shortcut toggles a breakpoint on the

    focused line in the LOG view.  Once breakpoints have

    been added, you can press `F7`/`F8` to move to the

    previous/next log message that matches a breakpoint.

  - The `:disable-breakpoint` and `:enable-breakpoint`

    commands have been added to allow disabling a

    breakpoint without deleting it.

  - If the log format specifies source file/line fields

    and a breakpoint is set, a red bullet point will be

    inserted to signify the presence of a breakpoint.

    Left-clicking on the bullet will toggle enabling/

    disabling the breakpoint.  A right-click will

    delete the breakpoint.

    In addition, if the `:add-source-path` command has

    been used, the first character of the source file

    will be underlined and can be left-clicked to open

    the source file at the given log message.  A

    right-click will set a breakpoint.

* The `all_opids` and `all_thread_ids` virtual tables

  have been added to make it simple to discover all of

  the operations and threads across all log files.  The

  `all_opids` table also supports setting a description

  for an operation using through an `UPDATE`.

* The `:xopen` command will now open text files in an

  external editor.  To open the file at a particular

  line/column, add a URL fragment of the form

  `L<line>C<column>`.

* When opening the contents of the prompt in an external

  editor (`CTRL+O`), the cursor position will be preserved,

  if possible.

* The `external-editor` configuration has been expanded

  with extra properties to help lnav choose the right one

  to use:

  - The `config-dir` property specifies the name of the

    directory that stores the editor's configuration in a

    source tree.  If the directory is found in an ancestor

    of the path to be opened, and it has the most recent

    modified time, the associated editor will be used.

  - The `prefers` property is a regular expression that

    will be tested against the full path to be opened.

    If matched, that editor will be chosen.

* The `:external-access` command has been added to open a

  localhost HTTP port that can be used to remotely control

  lnav. Requests can be sent to execute commands and poll

  for changes in the view state.  When the external port

  is open, a globe icon (🌐) is displayed in the top-right

  corner.  Clicking that icon will open a URL in a browser

  and log you into the server.  The `:external-access-login`

  command can also be used to login.

* Custom "Apps" can be added to the "external access"

  server to provide custom browser-based user interfaces

  to lnav.  See the "External Access" documentation online

  for more details.

* The `;.save` SQL command has been added that can save

  tables you have created to a SQLite database file.  The

  tables that lnav creates have been moved to a separate

  in-memory DB, so the main DB should only contain your

  own tables/views/etc...

* The `json_object_count_of()` SQL aggregate function has

  been added to make it easy to create a JSON object

  where the values are the number of times a value has

  been seen.

* The `:write-json-cols-to` command has been added to

  write JSON output in a column-oriented fashion.

* The `encode()` and `decode()` SQL functions now accept

  `html` as an algorithm.

* The `opid-field` can now be set to a JSON array/object

  for JSON-lines logs.  For example, the `spans` array in

  a Rust tracing log message.  The OPID for the message

  will be computed by hashing the contents of the array

  or object and the description will be the container

  itself.

* An OPID can now be constructed from multiple fields by

  leaving the `opid-field` blank and creating a single

  `opid/description` definition with a format.  The

  content of the format fields will then be hashed to

  create the OPID.  The builtin log formats have been

  updated to use this when appropriate.  For example,

  access_log now uses `c_ip` and `cs_user_agent` as the

  OPID.

  > [!NOTE]

  > If you want a description, but don't want it used as

  > the OPID.  You can set the `opid/source` field to

  > "from-whole-msg" and the OPID will be computed from

  > the contents of the log message.

* The `duration-field` log format property has been added

  to specify the field that contains a duration in the

  log message.  If a duration is available, it will be

  used to calculate time spans in the TIMELINE view.

  If the value of field is a number that is not in

  seconds, the `duration-divisor` property can be used

  to convert it.  For example, if the duration field is

  in milliseconds, the divisor should be 1000.  The

  duration can be accessed in the SQL vtables through

  the `log_duration` column.  For JSON-lines logs, the

  special `__duration__` field name can be used in the

  `line-format` to add a humanized version of the

  duration to the pretty-printed message.

* The TIMELINE view now shows rows for user-defined tags

  and partitions.  Each tagged log line appears as a

  separate entry in the timeline.  Tags whose names start

  or end with "start", "started", or "begin"

  (case-insensitive) will have their time range extended

  to the next instance of the same tag, with the last

  instance extending to the end of the log.  Partition

  rows span from their start time to the next partition

  (or end of log).

* Added `:hide-in-timeline` and `:show-in-timeline`

  commands to control which row types (logfile, thread,

  opid, tag, partition) are visible in the timeline view.

  The `:hide-in-timeline` command supports live preview,

  highlighting rows that would be hidden in red.

* The `timestamp-point-of-reference` log format property

  has been added to specify the relation of the timestamp

  to the operation that the message refers to, either:

  `start` or `end`.  This is used in conjunction with the

  message duration to determine the time span.

* The OPID for log messages is now shown in the parser

  details overlay (revealed by pressing `p`) in the

  LOG view.

* Added `rust_tracing_log` format from @richard-hajek.

* Added `macosuni_log` format that understands the

  output of the macOS `log stream --style=ndjson`

  command.

* Added the `idea_log` format from @segevfiner.

* The `strace_log` format has been improved to handle

  more output formats and the syscalls will now show

  up in the TIMELINE view.

* The `strace://` URL-handler has been added to make it

  easier to run `strace` on an existing process.  A

  host must be given and the path should be the PID,

  such as `strace://localhost/1234`.

* Added the `nestable` flag to the log format and

  theme highlight configurations to control whether a

  highlight can be applied to text that is covered by

  another highlight.

* Search tables are now included in the output of

  the `:export-session-to` command.

Breaking changes:

* All of lnav's SQLite tables have been moved to a

  separate in-memory database that is attached as

  `lnav_db`.  You may need to update some of your SQL

  statements to qualify table names with `lnav_db.`.

  This change was made so that the main DB only contains

  user data that can be easily backed up to a new DB

  file.

* Timestamp columns and results from lnav time functions

  now have microsecond precision instead of millisecond.

* The "module format" functionality has been removed.

  This functionality tried to match log messages wrapped

  in another format (usually syslog), but it never

  really worked well and was impeding progress in other

  areas.  Also, there have been many features added

  since the beginning that can serve the same use cases.

* The `sudo_log` format has been removed since it was

  a module-only format.  Instead, a `sudo_log`

  search-table was added to the `syslog_log` format.

Interface changes:

* Mouse mode is now enabled by default.

* The `CTRL+f` hotkey has been remapped to the

  `:toggle-filtering` command.

* Aborting the prompt now requires two successive

  presses of `Esc` (a message will pop up on the right

  that mentions this).  Since `Esc` is also used to

  close the completion popup, it was too easy to

  cancel the prompt.  Pressing `CTRL+]` will still

  close the prompt immediately.

* The TIMELINE view has a few updates:

  - The header has been redesigned to be one line that

    shows the time increments at the current scale.

    This approach should more clearly convey the spans

    of time shown in the main part of the view.  The

    previous design tried to show the overall time and

    the current time frame.  But, the multi-line header

    was hard to interpret and didn't make it clear how

    large the time increments were.

  - Log files and threads are now shown in the view

    in addition to operations.

* The HIST view now shows the year and inserts a spacer

  row in-between gaps in time.  The spacer row shows

  bullet points on a log scale to represent the amount

  of time in the gap.

* The SPECTRO view now shows the year in timestamps

  and uses additional colors to show the value range.

* The breadcrumb bar in the LOG view now includes the

  current thread, if defined.

* If there are background tasks, like the processing done

  by `:add-source-path`, a panel with progress bars for

  each operation will be shown just above the bottom

  status bar.

* The first line of a multi-line log message will now be

  shown in the header of the LOG view if the message has

  scrolled off the screen.  When scrolled to the

  beginning of the content, the top line will indicate

  if any log messages were filtered out before the first

  message.  Left-clicking on the header will scroll the

  view to the displayed line.

* In the Filters configuration panel, you can now

  create/edit the SQL expression filter by pressing the

  `e` key.

* The `{` hotkey and `:prev-section` command will now

  move to the first line of a multi-line log message if

  the focused line is in the middle of the message.

* The `}` hotkey and `:next-section` command will now

  move to the next log message if the focused line is

  in the middle of a multi-line message.

* The parser-details overlay now mentions any search

  tables that match the focused line.  Now, you don't

  have to remember the names of the tables.

* If a text file contains invalid UTF-8 content, the

  invalid bytes will be shown as the replacement

  character (�) and, when the line is focused, an

  overlay will show a hex dump of the line.

Performance:

* Searches now run in parallel using multiple child

  processes.  Large line ranges are split into chunks

  and distributed across available CPU cores, speeding

  up search in large files.

Bug Fixes:

* If a file path contains a hash (`#`), check if the path

  exists before splitting around the hash and treating it

  as an internal file location.

* The initialization sequence has been cleaned up to

  hopefully make it more consistent/reliable.  This

  concerns the sequence of loading files and executing

  commands from the command-line.

* In the SPECTRO view, you can now move to the next/

  previous row with bookmarks.  Note that this view

  synchronizes bookmarks with the LOG view.

* Session state should no longer override commands

  passed on the command-line.

* Marks in the TEXT view are now stable after filtering

  is applied.

* Error bookmarks are now added for stderr content when

  executing a command in the shell using `-e` or `:sh`.

Internal:

* Added operation IDs (OpIDs) to lnav's internal logging

  to make it easier to identify the high-level operations

  that are being performed using the TIMELINE view.

  While this is mainly useful to the authors of lnav, it

  can serve as a good example of the benefits of adding

  OpIDs and the TIMELINE view.

Thanks,

Tim Stack