openwisp-netconsole: real-time router kernel logs (netconsole) on the OpenWISP device page

[Michele Salerno]

Hi everyone,

I would like to share a small OpenWISP add-on I have just published:
openwisp-netconsole.

  https://github.com/mikysal78/openwisp-netconsole

WHAT IT DOES

It adds a "Netconsole" tab to the device page of the OpenWISP admin. The
tab shows, in real time, the kernel messages that the router sends with
Linux netconsole and that rsyslog stores on the server. It refreshes by
itself, asks the server only for the new lines and lets you download the
rotated log files.

It also adds a "Router log rotation" page (superusers only) to choose
how often the router logs are rotated, their maximum size and how long
they are kept, without editing configuration files on the server.

The repository also contains:
- an OpenWISP template for the devices: a hotplug script that loads
netconsole and tags every message with the device name;
- a basic Ansible playbook for the official ansible-openwisp2 role, with
the rsyslog configuration and the rotation script.

HOW IT WAS BORN

I run the OpenWISP instance of NINUX Basilicata, a community network in
southern Italy: a handful of OpenWrt routers (TP-Link, Linksys,
TOTOLINK, Zyxel and an x86 border router) connected by an 802.11s mesh,
cables and ZeroTier, with Babel routing on BIRD.

Some of these routers occasionally hung, and the remote syslog was useless:
when a router freezes or panics, syslog has no time to send anything. So
we started sending the kernel messages with netconsole, which works
until the very last moment, and collecting them with rsyslog.

That raised two problems:
1. netconsole has no hostname: the logs were stored by source IP, and a
router changes IP when it moves between LAN, WAN, mesh and VPN, so the
history of one router ended up in several folders;
2. to read them you had to log into the server, while everything else
about the device was already in OpenWISP.

For the first one, the router now adds the device name as netconsole
"userdata" (extended format), so rsyslog keeps one folder per device.
This needs a kernel with CONFIG_NETCONSOLE_DYNAMIC, which OpenWrt builds
as "n", so we enable it in our firmware build. For the second one, this
module shows the logs where you already look: on the device page.

WHY A SEPARATE PACKAGE

It is a small Django app that extends the device admin already
registered (including the openwisp-monitoring one) instead of replacing
it, so it can be added to an existing instance with two settings
(openwisp2_extra_python_packages and openwisp2_extra_django_apps).

Since it reads files on the server and the rotation is done by root, I
paid particular attention to security:
- only users who can see the device can read its logs, with the usual  
organization filtering;
- the browser never sends a path: it can only choose among the folders
computed by the server, symlinks are rejected and files are opened with
O_NOFOLLOW;
- a folder whose name or IP is shared with another device is not shown,
so logs cannot leak between devices or organizations;
- log text is inserted into the page as plain text only;
- OpenWISP never writes system configuration: the rotation page only
saves a small JSON file that a root script validates again (strict
ranges, fixed template) before generating the logrotate configuration.

The repository has CI (tests on Python 3.10-3.13, ruff, checks of the
example playbook and of the router script, rsyslog validation), CodeQL,
Dependabot,
a security policy for private vulnerability reporting, and releases with
a built wheel:

  pip install
https://github.com/mikysal78/openwisp-netconsole/releases/download/v0.1.1/openwisp_netconsole-0.1.1-py3-none-any.whl

It is still a 0.1 and it was born for our needs, so feedback, issues and
pull requests are very welcome. If you think something like this could
be useful in OpenWISP itself, I would be glad to discuss it.

Best regards,
Michele Salerno