Bug#770171: sshd jail fails when system solely relies on systemd journal for logging

[Tiziano Zito]

Package: fail2ban
Version: 0.9.1-1
Severity: important

Dear Maintainer,

when a system is configured to use the systemd journal as the
sole logging system, i.e. when none of the packages provided by
system-log-daemon are installed, the default sshd jail does not work.

When logging in the system is done by using the systemd journal, the
file /var/log/auth.log is not used anymore. While fail2ban 0.9
can use the systemd journal for matching offending log entries, the
Debian package comes with a "backend = auto" statement that
effectively disables matching against entries in the journal. As the
log files in /var/log are not updated anymore, fail2ban becomes
useless.

In order to have the sshd jail to work correctly I had to:

1. install python3-systemd, which is right now only suggested by
fail2ban, but given that systemd is going to be default in jessy
it should probably become a Depends

2. activate the systemd backend by adding
[DEFAULT]
backend = systemd
to the jail.d/defaults-debian.conf file

3. modify filter.d/sshd.conf to use the correct name of the sshd
systemd unit in Debian, which is ssh.service and not
sshd.service:
[Init]
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd

I did not find a way to perform 3 in a way that is robust against future
upgrades of the fail2ban package...

With the above mentioned modifications in place fail2ban correctly
bans abusive hosts. I am not sure if syslog-ng or rsyslog are still
going to be installed by default in jessy (probably yes?), but
I would assume that a number of people would want to solely rely on
the systemd journal, as otherwise logging gets duplicated and would
be unhappy to discover that fail2ban has not been working for months
(like it happened to me ;).

I don't know if fail2ban should use the systemd backend by default,
but the steps needed to make it work that way should be at least
mentioned in NEWS.Debian or README.Debian *and* the sshd filter
should use the correct name of the systemd unit [maybe all filters
should be checked for wrong systemd unit names?].

As a side note, do you think that package systemd should Provide
system-log-daemon? Is this worth filing a bug against systemd?

Ciao,
Tiziano

-- System Information:
Debian Release: jessie/sid
APT prefers unstable
APT policy: (500, 'unstable'), (101, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.17-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages fail2ban depends on:
ii init-system-helpers 1.21
ii lsb-base 4.1+Debian13+nmu1
ii python3 3.4.2-1
pn python3:any <none>

Versions of packages fail2ban recommends:
ii iptables 1.4.21-2+b1
pn python3-pyinotify <none>
ii whois 5.2.2

Versions of packages fail2ban suggests:
pn mailx <none>
ii python3-systemd 215-6
pn system-log-daemon <none>

-- Configuration Files:
/etc/fail2ban/filter.d/sshd.conf changed:
[INCLUDES]
before = common.conf
[Definition]
_daemon = sshd
failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error) for .* from <HOST>( via \S+)?\s*$
^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from <HOST>\s*$
^%(__prefix_line)sFailed \S+ for .*? from <HOST>(?: port \d*)?(?: ssh\d*)?(: (ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+ %(__md5hex)s(, client user ".*", client host ".*")?))?\s*$
^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$
^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*$
^%(__prefix_line)sUser .+ from <HOST> not allowed because listed in DenyUsers\s*$
^%(__prefix_line)sUser .+ from <HOST> not allowed because not in any group\s*$
^%(__prefix_line)srefused connect from \S+ \(<HOST>\)\s*$
^%(__prefix_line)sReceived disconnect from <HOST>: 3: \S+: Auth fail$
^%(__prefix_line)sUser .+ from <HOST> not allowed because a group is listed in DenyGroups\s*$
^%(__prefix_line)sUser .+ from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*$
^(?P<__prefix>%(__prefix_line)s)User .+ not allowed because account is locked<SKIPLINES>(?P=__prefix)(?:error: )?Received disconnect from <HOST>: 11: .+ \[preauth\]$
^(?P<__prefix>%(__prefix_line)s)Disconnecting: Too many authentication failures for .+? \[preauth\]<SKIPLINES>(?P=__prefix)(?:error: )?Connection closed by <HOST> \[preauth\]$
^(?P<__prefix>%(__prefix_line)s)Connection from <HOST> port \d+(?: on \S+ port \d+)?<SKIPLINES>(?P=__prefix)Disconnecting: Too many authentication failures for .+? \[preauth\]$
ignoreregex =
[Init]
maxlines = 10
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd

/etc/fail2ban/jail.d/defaults-debian.conf changed:
[DEFAULT]
backend = systemd
[sshd]
enabled = true


-- no debconf information


--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

[Jan Heitkötter]

Install rsyslog? Nonsense.

Creating /etc/fail2ban/jail.local with


[DEFAULT]

default_backend = systemd


will do the trick.

IMO the error lies within /etc/fail2ban/paths-common.conf which says


[DEFAULT]

default_backend = auto


This should be changed to


default_backend = systemd

[Sebastian Walther]

hi there,

i surprisingly discovered today that fail2ban didn't work for quite some (too long) time. Reason was changing from syslog to systemd. after changing "backend = auto" to "backend = systemd" and wondering why it fail all the time, it took me hours to find out that there is a bug/ missing dependency.

guys please fix this! this bug is nearly 3 years old!! or is there any technical reason for this?

autor has already mentioned how to fix this.

1. add python[1-3]-systemd  as dependency

2. fix /etc/failt2ban/filter.d/sshd.conf  

            form

                        journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd

            to

                        journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd

btw

duplicated bug here:         https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862348

pls fix :)

[SZÉPE Viktor]

Idézem/Quoting Sebastian Walther <sebastian...@googlemail.com>:

> hi there,
>
> i surprisingly discovered today that fail2ban didn't work for quite
> some (too long) time. Reason was changing from syslog to systemd.
> after changing "backend = auto" to "backend = systemd" and wondering
> why it fail all the time, it took me hours to find out that there is
> a bug/ missing dependency.
>
>
> guys please fix this! this bug is nearly 3 years old!! or is there
> any technical reason for this?

This is not a solution but
you may try to install Fail2ban from testing
https://packages.qa.debian.org/f/fail2ban.html
It is quite old but may help you.



SZÉPE Viktor, honlap üzemeltetés
https://github.com/szepeviktor/debian-server-tools/blob/master/CV.md
--
+36-20-4242498 s...@szepe.net skype: szepe.viktor
Budapest, III. kerület

[Vincent Van Houtte]

I can confirm this behaviour is still buggy in version 0.9.6-2 (current debian stretch repo). So far I have switched to systemd backends for sshd and postfix on my machine, and I had to make the following changes:

replace

journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd

with

journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd

in

/etc/fail2ban/filter.d/sshd.conf

/etc/fail2ban/filter.d/sshd-ddos.conf

and replace

journalmatch = _SYSTEMD_UNIT=postfix.service

with

journalmatch = _SYSTEMD_UNIT=postfix@-.service

in

/etc/fail2ban/filter.d/postfix.conf

/etc/fail2ban/filter.d/postfix-sasl.conf

HTH,

Vincent

[von Obernitz, Daniel]

Hi,

in Debian 11 with f2b version 0.11.2-2 the issue with "journalmatch" seems to be fixed, but now the filter is wrong.

/etc/fail2ban/filter.d/sshd.conf

# consider failed publickey for invalid users only:

cmnfailre-failed-pub-invalid = ^Failed publickey for invalid user <F-USER>(?P<cond_user>\S+)|(?:(?! from ).)*?</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)

The log output in systemd does not match, the regex has to be changed to something like:

cmnfailre-failed-pub-invalid = ^Failed publickey for <F-USER>(?P<cond_user>\S+)|(?:(?! from ).)*?</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)


Daniel

[Nathaniel Mitchell]

Hi all,

Still an issue for the backend option being left on 'auto', could the default for Debian be changed to use systemd as the backend?

(Fresh install on Bookworm).

Regards,

 

Nephi.Aust

[Andrei Coada]

Hi Team,

This is getting pretty annoying, a 9 years old inconvenience, especially now that Debian 12 does not even have a syslog service installed by default.

Fail2ban fails to start right after its installation.

The solution is really trivial. At least an SSHD override should be added in  "paths-debian.conf"  file, such as:

sshd_backend = systemd

so that the fail2ban service can start after one installs it.

Thanks,
Andrei

[Richard Lewis]

On Fri, 11 Aug 2023 01:11:49 +0100 Andrei Coada <andrei...@gmail.com> wrote:
> Hi Team,
>
> This is getting pretty annoying, a 9 years old inconvenience, especially
> now that Debian 12 does not even have a syslog service installed by default.
> Fail2ban fails to start right after its installation.

did you  read the esrlier messages in the bug report  - those messages say that fail2ban works provided python3-systemd is installed. there is no need to edit any config files.

[Andrei Coada]

Hi Richard,

Thanks for your swift reply.

python3-systemd is just a wrapper that ties python to systemd's functions. How would one expect it to magically solve this issue if no one _instructs_ the SSHD jail to actually use the logs in the journal instead of some file logs(which is the default setting)?

Reproduction:

default  paths_debian.conf  contents

# dpkg -l | grep python3-systemd
ii  python3-systemd               235-1+b2                       amd64        Python 3 bindings for systemd

root@blabla:/etc/fail2ban# service fail2ban start
root@blabla:/etc/fail2ban# service fail2ban status
× fail2ban.service - Fail2Ban Service
    Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; preset: enabled)
    Active: failed (Result: exit-code) since Fri 2023-08-11 15:14:52 EEST; 5s ago
  Duration: 123ms
      Docs: man:fail2ban(1)
   Process: 91337 ExecStart=/usr/bin/fail2ban-server -xf start (code=exited, status=255/EXCEPTION)
  Main PID: 91337 (code=exited, status=255/EXCEPTION)
       CPU: 120ms
Aug 11 15:14:52 seif.radiocom.ro systemd[1]: Started fail2ban.service - Fail2Ban Service.
Aug 11 15:14:52 seif.radiocom.ro fail2ban-server[91337]: 2023-08-11 15:14:52,937 fail2ban                [91337]: ERROR   Failed during configuration: Have not found any log file for sshd jail
Aug 11 15:14:52 seif.radiocom.ro fail2ban-server[91337]: 2023-08-11 15:14:52,940 fail2ban                [91337]: ERROR   Async configuration of server failed

There are at least 2 options I can think of to overcome this issue:

1. Add  sshd_backend = systemd  to  paths_debian.conf

2. Create a  jail.local  file with the following contents:

[sshd]
backend = systemd
enabled = true

Then it'll work.

In conclusion, just installing  python3-systemd  package _won't_ fix anything, therefore this should still be considered a bug, as this software does _NOT_ work out-of-the-box.

Thanks,
Andrei

[Andrei Coada]

Hi again,

Also, it seems to me that you've skipped some reading from the original bug report. I quote Tiziano Zito, saying:

"In order to have the sshd jail to work correctly I had to:

1. install python3-systemd, which is right now only suggested by
   fail2ban, but given that systemd is going to be default in jessy
   it should probably become a Depends

2. activate the systemd backend by adding 
   [DEFAULT]
   backend = systemd 

   to the jail.d/defaults-debian.conf file"

So switching to the systemd backend was there from the beginning.

Thanks,
Andrei

[Stefan Weil]

This bug report from 2014 is meanwhile more important than ever.

With the latest stable release Debian marked rsyslog as deprecated (see
https://wiki.debian.org/Rsyslog).

If a user removes the rsyslog package without removing the related
logfiles in /var/log, fail2ban silently stops doing its job, because
it looks for failed ssh logins in /var/log/auth which no longer gets
updates. The same applies to other jails which were activated by local
settings.

If the user not only removes rsyslog but also removes all old files in
/var/log/, fail2ban no longer runs at all but silently fails.

If fail2ban is not working as expected or not running at all, that can
effect the security of a Debian system. Therefore I suggest to increase
the priority of this bug report.

My fix for this issue (and also for issue 1024305) is a small
modification of jail.d/defaults-debian.conf:

------

# cat /etc/fail2ban/jail.d/defaults-debian.conf
[DEFAULT]
allowipv6 = auto
dovecot_backend = systemd
postfix_backend = systemd
sshd_backend = systemd
# ... add more affected backends here

[sshd]
enabled = true

------

I only added the backends which where required for my Debian system.
Maybe some other jails also must use the systemd backend.

I suggest to update jail.d/defaults-debian.conf and distribute that as a
security fix.

In addition, rsyslog could be removed from the list of suggested
packages in future Debian releases.

Stefan

[Stefaan Vanheesbeke]

Confirmed that this is still annoying. I (as a moderate linux user) had
to search several hours, during several days until finding the fix.

It works fine in a debian11 fresh install, not in a debian12. So I
suppose it was fixed some time before, but came back an issue recently.

After adding "backend=systemd" to the jail.d/defaults-debian.conf in the
ssh section should be the simplest solution for having back some safety
in a default apt-get install.

An extra burden is that you need to manually install python3-systemd
also. If not, it will not work after a fresh install. This should be a
dependency on the jail2ban installation.

For me, it is not the extra work to do to make it work that is the
problem, the big issue is that people may rely on the fact that they
installed fail2ban, and think their safety increased, however it doesn't
work out of the box! I wonder how they make all these tutorials.

Have a nice day,

Stefaan

[Luca Capello]

found 770171 1.0.2-2
tags 770171 + upstream
forwarded 770171 https://github.com/fail2ban/fail2ban/issues/1372
user lu...@pca.it
usertags 770171 + pca.it-security
thanks

Hi there,

I just got hit by this as well, plus the nftables issue:

<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1049452#10>

On Tue, 02 Jan 2024 19:39:13 +0000, Debian Bug Tracking System wrote:
> Changes:
> fail2ban (1.0.2-3) unstable; urgency=medium
> .
> * Add banaction = nftables in the defaults-debian.conf default
> see https://github.com/fail2ban/fail2ban/discussions/3575#discussioncomment-7045315
> * Move python3-systemd as depend (Closes: #770171, #1037437)
> * Add backend = systemd to jail.d/defaults-debian.conf

The commit corresponding to the last line is:

<https://salsa.debian.org/python-team/packages/fail2ban/-/blob/e634fa863e2d8035181e9e03476ae6dd56044fe6/debian/debian-files/jail.d_defaults-debian.conf>

For those waiting the fixed pacakge, without touching any other
packaged file I can confirm that `apt-get install python3-systemd`
plus the below `jail.local` are enough to get `fail2ban` starts on a
fresh Debian 12.4/bookworm:
```
[DEFAULT]
### <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1049452#10>
banaction = nftables
banaction_allports = nftables[type=allports]
### <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770171#94>
backend = systemd
```

Thx, bye,
Gismo / Luca

PS, the upstream bug is quite interesting, especially to understand the
differences between `(|*_)backend`:

<https://github.com/fail2ban/fail2ban/issues/1372>