systemd-journal-remote: Does it Work?

[joeb...@gmail.com]

Hi all, 

I'm having trouble setting up log aggregation with systemd-journal-remote on CoreOS 835.11.0.

When I try to start the service, I'm getting this error:

# systemctl status systemd-journal-remote

● systemd-journal-remote.service - Journal Remote Sink Service

   Loaded: loaded (/usr/lib64/systemd/system/systemd-journal-remote.service; indirect; vendor preset: disabled)

   Active: failed (Result: exit-code) since Sun 2016-01-24 21:03:24 UTC; 32s ago

  Process: 15760 ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-https=-3 --output=/var/log/journal/remote/ (code=exited, status=1/FAILURE)

 Main PID: 15760 (code=exited, status=1/FAILURE)

Jan 24 21:03:24 srv-03 systemd[1]: Started Journal Remote Sink Service.

Jan 24 21:03:24 srv-03 systemd-journal-remote[15760]: Failed to start µhttp daemon

Jan 24 21:03:24 srv-03 systemd-journal-remote[15760]: Failed to register socket (fd:3): Invalid argument

The socket seems to be okay from what I can gather:

# systemctl status systemd-journal-remote.socket

● systemd-journal-remote.socket - Journal Remote Sink Socket

   Loaded: loaded (/usr/lib64/systemd/system/systemd-journal-remote.socket; enabled; vendor preset: disabled)

   Active: active (listening) since Sun 2016-01-24 21:02:52 UTC; 58s ago

   Listen: [::]:19532 (Stream)

Jan 24 21:02:52 srv-03 systemd[1]: Listening on Journal Remote Sink Socket.

Similar error when I try the command line:

# /usr/lib/systemd/systemd-journal-remote --listen-https=19532 --output=/var/log/journal/remote/ --cert=/etc/ssl/private/journal-remote.pem

Failed to start µhttp daemon

In addition, if I reboot the computer, it loses the permissions on /etc/ssl/private folder, so the service then fails with permission denied:

# ls -la /etc/ssl/private

drwx-----x 2 root root 4096 Jan 24 20:14 .

drwxr-xr-x 6 root root 4096 Jan 24 20:11 ..

-rw-r--r-- 1 root root  798 Jan 24 20:14 journal-remote.pem

# reboot now 

...

# ls -la /etc/ssl/private/

drwx------ 2 root root 4096 Jan 24 20:14 .

This leads to the service failing to start with a permission denied error:

● systemd-journal-remote.service - Journal Remote Sink Service

   Loaded: loaded (/usr/lib64/systemd/system/systemd-journal-remote.service; indirect; vendor preset: disabled)

   Active: failed (Result: exit-code) since Sun 2016-01-24 21:26:22 UTC; 2s ago

  Process: 1008 ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-https=-3 --output=/var/log/journal/remote/ (code=exited, status=1/FAILURE)

 Main PID: 1008 (code=exited, status=1/FAILURE)

Jan 24 21:26:22 srv-03 systemd[1]: Started Journal Remote Sink Service.

Jan 24 21:26:22 srv-03 systemd-journal-remote[1008]: Failed to read key from file '/etc/ssl/private/journal-remote.pem': Permission denied

I've googled a bit, and there are a few confusing discussions on the bug tracker; it is unclear to me from the conversations there whether this has all been resolved and tested or not.

Has anyone gotten this service working on CoreOS?

Thanks,

Jonathan

[joeb...@gmail.com]

To answer my own question: it appears that most of the problems I ran into were fixed in the current beta, 899.5.0.

[anton....@coreos.com]

Dear Jonathan,

Unfortunately systemd-journal-remote doesn't work out of the box yet. Here are related tickets:  https://github.com/coreos/bugs/issues/919 https://github.com/coreos/bugs/issues/962

And unfortunately "https" doesn't work in CoreOS because CoreOS doesn't ship gnutls library. So you have to use "--listen-http=-3" parameter.

You can set insecure http connection using drop-in (https://coreos.com/os/docs/latest/using-systemd-drop-in-units.html):

[Service]

ExecStart=

ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/

But if it is required for you to use https for security reasons you can compile your own libmicrohttpd library with statically linked gnutls, copy it into CoreOS's host (/opt/lib) and use LD_LIBRARY_PATH=/opt/lib environment variable in "systemd-journal-remote" drop-in.

Let me know if you have any further questions.

Regards,

Anton

[joeb...@gmail.com]

Hi Anton,

many thanks for taking the time to reply in detail.

I gathered as much as you said, unencrypted log replication is okay for now as I'm running it over a VLAN.

Could not figure it out from the referenced issues: is this a won't fix thing, or do you intend to fix that issue with https transport?

Thanks,

Jon

[anton....@coreos.com]

Dear Jonathan,

This question is still open. But I'm not sure that it will be fixed soon. We don't want to ship a bunch of SSL libs, and it is not easy to port libmicrohttpd to use openssl.

Regards,

Anton