Bug#1054391: systemd-journal-remote aborts when getting remote messages complaining about field size

[jdehaan]

Package: systemd-journal-remote
Version: 252.17-1~deb12u1
Severity: normal
Tags: upstream
X-Debbugs-Cc: jde...@zwartkasteel.nl




-- System Information:
Debian Release: 12.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-13-amd64 (SMP w/12 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages systemd-journal-remote depends on:
ii libc6 2.36-9+deb12u3
ii libcurl4 7.88.1-10+deb12u4
ii libmicrohttpd12 0.9.75-6
ii libsystemd-shared 252.17-1~deb12u1
ii systemd 252.17-1~deb12u1

systemd-journal-remote recommends no packages.

systemd-journal-remote suggests no packages.

-- no debconf information


On connecting a remote (debian 12.2 named 'source') system to upload it's
journal messages to a central journal system (named 'sink') the
following error messages were produced:

Oct 22 15:49:35 source systemd-journal-upload[11706]: Upload to http://sinkip:19532/upload failed: Recv failure: Connection reset by peer
Oct 22 15:49:35 source systemd[1]: systemd-journal-upload.service: Main process exited, code=exited, status=1/FAILURE
Oct 22 15:49:35 source systemd[1]: systemd-journal-upload.service: Failed with result 'exit-code'.
Oct 22 15:49:35 source systemd[1]: systemd-journal-upload.service: Consumed 4.256s CPU time.

Oct 22 17:02:44 source systemd-journal-upload[12136]: Upload to http://sinkip:19532/upload failed: Recv failure: Connection reset by peer
Oct 22 17:02:44 source systemd[1]: systemd-journal-upload.service: Main process exited, code=exited, status=1/FAILURE
Oct 22 17:02:44 source systemd[1]: systemd-journal-upload.service: Failed with result 'exit-code'.
Oct 22 17:02:44 source systemd[1]: systemd-journal-upload.service: Consumed 4.340s CPU time.

Oct 22 15:49:27 sink systemd-journal-remote[1397]: /var/log/journal/remote//remote-sourceip.journal: Journal header limits reached or header out-of-date, rotating
Oct 22 15:49:29 sink systemd-journal-remote[1397]: /var/log/journal/remote//remote-sourceip.journal: Journal header limits reached or header out-of-date, rotating
Oct 22 15:49:32 sink systemd-journal-remote[1397]: Entry with no payload, skipping
Oct 22 15:49:35 sink systemd-journal-remote[1397]: /var/log/journal/remote//remote-sourceip.journal: Journal header limits reached or header out-of-date, rotating
Oct 22 15:49:35 sink systemd-journal-remote[1397]: Stream declares field with size 4922226994454288453 > DATA_SIZE_MAX = 805306368

Oct 22 17:02:36 sink systemd-journal-remote[1397]: /var/log/journal/remote//remote-sourceip.journal: Journal header limits reached or header out-of-date, rotating
Oct 22 17:02:38 sink systemd-journal-remote[1397]: /var/log/journal/remote//remote-sourceip.journal: Journal header limits reached or header out-of-date, rotating
Oct 22 17:02:41 sink systemd-journal-remote[1397]: Entry with no payload, skipping
Oct 22 17:02:44 sink systemd-journal-remote[1397]: /var/log/journal/remote//remote-sourceip.journal: Journal header limits reached or header out-of-date, rotating
Oct 22 17:02:44 sink systemd-journal-remote[1397]: Stream declares field with size 4922226994454288453 > DATA_SIZE_MAX = 805306368

If needed more information can be provided.

[Jan de Haan]

L.S.,

this turned out to be something really specific.

https://github.com/systemd/systemd/issues/29676

Root cause found, at least: as far as I can dig.

The system running the sink is connected to two networks and is accepting journals from 1 system on one and 6 on the other network. All systems have the above mentioned versions (Debian 12.2, kernel 6.1.0-13-amd64, systemd 252.17-1~deb12u1), all are using the systemd-journal-upload.

The six on the first network were working fine, dumping to their hearts content.

The single system on the second was having the above mentioned problems.

Working with precision, the systemd-journal-remote.socket's ListenStreams were configured to be listening on the ip addresses reserved for that specific purpose. This was done with the edit/override possibility on the socket:

$ sudo systemctl edit systemd-journal-remote.socket :
...
[Socket]
ListenStream=
ListenStream=10.0.0.202:19532
ListenStream=10.1.1.202:19532
...

Guess what: the 6 working clients were on the first ip, the 1 that didn't on the second.

• Changing the order resulted in 1 working and 6 failing clients.
• Adding an additional ListenStream with 127.0.0.1 didn't change anything.
• Removing all ListenStream settings resulting in listening on 0.0.0.0:19532 made all 7 systems working.

JdH.

--

"Piracy is simply demand where supply does not exist."