Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#860974: lxc: systemd and other inotify-based programs fail with "Too many open files" with a large number of LXC containers
209 views
Skip to first unread message
Baptiste Jonglez
Apr 22, 2017, 7:10:02 PM4/22/17
to
Package: lxc
Version: 1:2.0.7-2
Severity: important
Tags: patch
Dear Maintainer,
I am using LXC on stretch, running a number of stretch containers.
When running around 15 containers, systemd starts to fail, both on the host
and inside containers:
# journalctl -f
Failed to get journal fd: Too many open files
# systemctl stop nginx
Failed to allocate directory watch: Too many open files
The problem seems to be that systemd uses inotify quite a lot, and the limit
on inotify listeners is very low by default:
# sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 128
Increasing this value indeed solves the problem. The issue was described
in this blog post:
https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
Attached is a patch for lxc that configures this sysctl setting to a higher
value. I don't know if this is the "Debian way" or if some other solution
is preferred.
-- System Information:
Debian Release: 9.0
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64
(x86_64)
Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages lxc depends on:
ii init-system-helpers 1.47
ii libapparmor1 2.11.0-3
ii libc6 2.24-9
ii libcap2 1:2.25-1
ii libgnutls30 3.5.8-5
ii liblxc1 1:2.0.7-2
ii libseccomp2 2.3.1-2.1
ii libselinux1 2.6-3+b1
ii lsb-base 9.20161125
ii python3-lxc 1:2.0.7-2
pn python3:any <none>
Versions of packages lxc recommends:
pn bridge-utils <none>
ii debootstrap 1.0.89
ii dirmngr 2.1.18-6
pn dnsmasq-base <none>
ii gnupg 2.1.18-6
ii iptables 1.6.0+snapshot20161117-6
pn libpam-cgfs <none>
pn lxcfs <none>
ii openssl 1.1.0e-1
ii rsync 3.1.2-1
pn uidmap <none>
Versions of packages lxc suggests:
pn apparmor <none>
pn btrfs-tools <none>
ii lvm2 2.02.168-2
-- no debconf information
Version: 1:2.0.7-2
Severity: important
Tags: patch
Dear Maintainer,
I am using LXC on stretch, running a number of stretch containers.
When running around 15 containers, systemd starts to fail, both on the host
and inside containers:
# journalctl -f
Failed to get journal fd: Too many open files
# systemctl stop nginx
Failed to allocate directory watch: Too many open files
The problem seems to be that systemd uses inotify quite a lot, and the limit
on inotify listeners is very low by default:
# sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 128
Increasing this value indeed solves the problem. The issue was described
in this blog post:
https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
Attached is a patch for lxc that configures this sysctl setting to a higher
value. I don't know if this is the "Debian way" or if some other solution
is preferred.
-- System Information:
Debian Release: 9.0
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64
(x86_64)
Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages lxc depends on:
ii init-system-helpers 1.47
ii libapparmor1 2.11.0-3
ii libc6 2.24-9
ii libcap2 1:2.25-1
ii libgnutls30 3.5.8-5
ii liblxc1 1:2.0.7-2
ii libseccomp2 2.3.1-2.1
ii libselinux1 2.6-3+b1
ii lsb-base 9.20161125
ii python3-lxc 1:2.0.7-2
pn python3:any <none>
Versions of packages lxc recommends:
pn bridge-utils <none>
ii debootstrap 1.0.89
ii dirmngr 2.1.18-6
pn dnsmasq-base <none>
ii gnupg 2.1.18-6
ii iptables 1.6.0+snapshot20161117-6
pn libpam-cgfs <none>
pn lxcfs <none>
ii openssl 1.1.0e-1
ii rsync 3.1.2-1
pn uidmap <none>
Versions of packages lxc suggests:
pn apparmor <none>
pn btrfs-tools <none>
ii lvm2 2.02.168-2
-- no debconf information
Evgeni Golov
Apr 23, 2017, 5:50:03 AM4/23/17
to
Hi,
thanks for the report!
On Sun, Apr 23, 2017 at 12:31:02AM +0200, Baptiste Jonglez wrote:
> I am using LXC on stretch, running a number of stretch containers.
>
> When running around 15 containers, systemd starts to fail, both on the host
> and inside containers:
>
> # journalctl -f
> Failed to get journal fd: Too many open files
> # systemctl stop nginx
> Failed to allocate directory watch: Too many open files
>
> The problem seems to be that systemd uses inotify quite a lot, and the limit
> on inotify listeners is very low by default:
>
> # sysctl fs.inotify.max_user_instances
> fs.inotify.max_user_instances = 128
>
> Increasing this value indeed solves the problem. The issue was described
> in this blog post:
>
> https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
>
> Attached is a patch for lxc that configures this sysctl setting to a higher
> value. I don't know if this is the "Debian way" or if some other solution
> is preferred.
Your patch looks quite similar to what was added in LXD in Ubuntu recently:
https://github.com/lxc/lxd-pkg-ubuntu/commit/333e2671a91861b949a2b990459656a3b065e5fb
https://bugs.launchpad.net/juju/+bug/1602192
I think we also should add something like
sysctl -q -p /etc/sysctl.d/lxc.conf || true
to the configure target of postinst, to actually load the file directly,
instead of waiting for a reboot?
However, I am not 100% sure that packages should alter sysctl settings without
telling the admin.
Regards
Evgeni
thanks for the report!
On Sun, Apr 23, 2017 at 12:31:02AM +0200, Baptiste Jonglez wrote:
> I am using LXC on stretch, running a number of stretch containers.
>
> When running around 15 containers, systemd starts to fail, both on the host
> and inside containers:
>
> # journalctl -f
> Failed to get journal fd: Too many open files
> # systemctl stop nginx
> Failed to allocate directory watch: Too many open files
>
> The problem seems to be that systemd uses inotify quite a lot, and the limit
> on inotify listeners is very low by default:
>
> # sysctl fs.inotify.max_user_instances
> fs.inotify.max_user_instances = 128
>
> Increasing this value indeed solves the problem. The issue was described
> in this blog post:
>
> https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
>
> Attached is a patch for lxc that configures this sysctl setting to a higher
> value. I don't know if this is the "Debian way" or if some other solution
> is preferred.
https://github.com/lxc/lxd-pkg-ubuntu/commit/333e2671a91861b949a2b990459656a3b065e5fb
https://bugs.launchpad.net/juju/+bug/1602192
I think we also should add something like
sysctl -q -p /etc/sysctl.d/lxc.conf || true
to the configure target of postinst, to actually load the file directly,
instead of waiting for a reboot?
However, I am not 100% sure that packages should alter sysctl settings without
telling the admin.
Regards
Evgeni
Baptiste Jonglez
Apr 23, 2017, 7:20:03 AM4/23/17
to
Hi Evgeni,
On Sun, Apr 23, 2017 at 11:45:07AM +0200, Evgeni Golov wrote:
> > Attached is a patch for lxc that configures this sysctl setting to a higher
> > value. I don't know if this is the "Debian way" or if some other solution
> > is preferred.
>
> Your patch looks quite similar to what was added in LXD in Ubuntu recently:
> https://github.com/lxc/lxd-pkg-ubuntu/commit/333e2671a91861b949a2b990459656a3b065e5fb
> https://bugs.launchpad.net/juju/+bug/1602192
Ah, indeed, I had found the launchpad bug but not the patch.
> I think we also should add something like
> sysctl -q -p /etc/sysctl.d/lxc.conf || true
> to the configure target of postinst, to actually load the file directly,
> instead of waiting for a reboot?
Good idea!
> However, I am not 100% sure that packages should alter sysctl settings without
> telling the admin.
Well, that was the meaning of my question about the "Debian way" :)
The only two packages I found doing this are bit-babbler and ufw, so it
does not exactly seem like a widespread practice.
Baptiste
On Sun, Apr 23, 2017 at 11:45:07AM +0200, Evgeni Golov wrote:
> > Attached is a patch for lxc that configures this sysctl setting to a higher
> > value. I don't know if this is the "Debian way" or if some other solution
> > is preferred.
>
> Your patch looks quite similar to what was added in LXD in Ubuntu recently:
> https://github.com/lxc/lxd-pkg-ubuntu/commit/333e2671a91861b949a2b990459656a3b065e5fb
> https://bugs.launchpad.net/juju/+bug/1602192
> I think we also should add something like
> sysctl -q -p /etc/sysctl.d/lxc.conf || true
> to the configure target of postinst, to actually load the file directly,
> instead of waiting for a reboot?
> However, I am not 100% sure that packages should alter sysctl settings without
> telling the admin.
The only two packages I found doing this are bit-babbler and ufw, so it
does not exactly seem like a widespread practice.
Baptiste
0 new messages