Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1042807: acng becomes unresponsive on high load
121 views
Skip to first unread message
Harald Dunkel
Aug 1, 2023, 3:20:05 AM8/1/23
to
Package: apt-cacher-ng
Version: 3.7.4-1+b2
If I try to connect upgrade a few (24) virtual hosts in parallel
(using apt-cacher-ng as a cache), then 5 or 10% might succeed
on "apt update", but the rest fails with something like
# apt update
Get:1 file:/var/cache/apt-build/repository apt-build InRelease
Ign:1 file:/var/cache/apt-build/repository apt-build InRelease
Get:2 file:/var/cache/apt-build/repository apt-build Release [1530 B]
Get:2 file:/var/cache/apt-build/repository apt-build Release [1530 B]
Get:3 file:/var/cache/apt-build/repository apt-build Release.gpg
Ign:3 file:/var/cache/apt-build/repository apt-build Release.gpg
Err:4 https://apt.kubernetes.io kubernetes-xenial InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:5 https://pkg.jenkins.io/debian binary/ InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:6 https://packages.cloud.google.com/apt cloud-sdk InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Hit:7 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:8 http://deb.debian.org/debian bullseye InRelease
Err:9 https://baltocdn.com/helm/stable/debian all InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:10 https://download.docker.com/linux/debian bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:11 https://packages.adoptium.net/artifactory/deb bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:12 https://packages.gitlab.com/runner/gitlab-runner/debian bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:13 https://packages.microsoft.com/repos/azure-cli bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:14 https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/debs vscodium InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:15 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:16 http://security.debian.org bullseye-security InRelease
Connection failed [IP: 172.19.96.126 3142]
Err:17 http://debian.example.com/debian bullseye-backports InRelease
Connection failed [IP: 172.19.96.126 3142]
Err:18 https://packages.gitlab.com/gitlab/gitlab-ee/debian bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:19 http://packages.microsoft.com/repos/code stable InRelease
Connection failed [IP: 172.19.96.126 3142]
Err:20 https://packages.microsoft.com/repos/ms-teams stable InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:21 http://download.virtualbox.org/virtualbox/debian bullseye InRelease
Connection failed [IP: 172.19.96.126 3142]
Hit:22 http://debian.example.com/debian bullseye InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://security.debian.org/dists/bullseye-security/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.adoptium.net/artifactory/deb/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch http://debian.example.com/debian/dists/bullseye-backports/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://baltocdn.com/helm/stable/debian/dists/all/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://download.docker.com/linux/debian/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://pkg.jenkins.io/debian/binary/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.gitlab.com/runner/gitlab-runner/debian/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ee/debian/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.microsoft.com/repos/azure-cli/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.microsoft.com/repos/ms-teams/dists/stable/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch http://download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch http://packages.microsoft.com/repos/code/dists/stable/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/debs/dists/vscodium/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.
acng's error log shows
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for debian.example.com/debian/dists/bullseye-backports/InRelease -- check file permissions!
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for debian.example.com/debian/dists/bullseye-backports/InRelease -- check file permissions!
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for debian.example.com/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/packages.microsoft.com/repos/code/dists/stable/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/packages.microsoft.com/repos/code/dists/stable/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for packages.microsoft.com/repos/code/dists/stable/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/secdeb/dists/bullseye-security/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/secdeb/dists/bullseye-security/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for secdeb/dists/bullseye-security/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|/var/cache/apt-cacher-ng/debrep/dists/bullseye/InRelease.1690871025 storage error [Checked size beyond EOF], check file AND directory permissions, last errno: File exists
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/dl.google.com/linux/chrome/deb/dists/stable/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/dl.google.com/linux/chrome/deb/dists/stable/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for dl.google.com/linux/chrome/deb/dists/stable/InRelease -- check file permissions!
Tue Aug 1 08:24:16 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease.1690871056 - errno: File exists
Tue Aug 1 08:24:16 2023|Error creating file item for debian.example.com/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:24:16 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease.1690871056 - errno: File exists
Tue Aug 1 08:24:16 2023|Error creating file item for debian.example.com/debian/dists/bullseye/InRelease -- check file permissions!
limits.conf is set to
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 16584
root hard nofile 16584
I cannot remember similar problems for Debian 10. Would it be possible
to make acng more reliable under heavy load?
Regards
Harri
Version: 3.7.4-1+b2
If I try to connect upgrade a few (24) virtual hosts in parallel
(using apt-cacher-ng as a cache), then 5 or 10% might succeed
on "apt update", but the rest fails with something like
# apt update
Get:1 file:/var/cache/apt-build/repository apt-build InRelease
Ign:1 file:/var/cache/apt-build/repository apt-build InRelease
Get:2 file:/var/cache/apt-build/repository apt-build Release [1530 B]
Get:2 file:/var/cache/apt-build/repository apt-build Release [1530 B]
Get:3 file:/var/cache/apt-build/repository apt-build Release.gpg
Ign:3 file:/var/cache/apt-build/repository apt-build Release.gpg
Err:4 https://apt.kubernetes.io kubernetes-xenial InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:5 https://pkg.jenkins.io/debian binary/ InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:6 https://packages.cloud.google.com/apt cloud-sdk InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Hit:7 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:8 http://deb.debian.org/debian bullseye InRelease
Err:9 https://baltocdn.com/helm/stable/debian all InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:10 https://download.docker.com/linux/debian bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:11 https://packages.adoptium.net/artifactory/deb bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:12 https://packages.gitlab.com/runner/gitlab-runner/debian bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:13 https://packages.microsoft.com/repos/azure-cli bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:14 https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/debs vscodium InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:15 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:16 http://security.debian.org bullseye-security InRelease
Connection failed [IP: 172.19.96.126 3142]
Err:17 http://debian.example.com/debian bullseye-backports InRelease
Connection failed [IP: 172.19.96.126 3142]
Err:18 https://packages.gitlab.com/gitlab/gitlab-ee/debian bullseye InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:19 http://packages.microsoft.com/repos/code stable InRelease
Connection failed [IP: 172.19.96.126 3142]
Err:20 https://packages.microsoft.com/repos/ms-teams stable InRelease
Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
Err:21 http://download.virtualbox.org/virtualbox/debian bullseye InRelease
Connection failed [IP: 172.19.96.126 3142]
Hit:22 http://debian.example.com/debian bullseye InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://security.debian.org/dists/bullseye-security/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.adoptium.net/artifactory/deb/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch http://debian.example.com/debian/dists/bullseye-backports/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://baltocdn.com/helm/stable/debian/dists/all/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://download.docker.com/linux/debian/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://pkg.jenkins.io/debian/binary/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.gitlab.com/runner/gitlab-runner/debian/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ee/debian/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.microsoft.com/repos/azure-cli/dists/bullseye/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch https://packages.microsoft.com/repos/ms-teams/dists/stable/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Failed to fetch http://download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch http://packages.microsoft.com/repos/code/dists/stable/InRelease Connection failed [IP: 172.19.96.126 3142]
W: Failed to fetch https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/debs/dists/vscodium/InRelease Reading from proxy failed - select (115: Operation now in progress) [IP: 172.19.96.126 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.
acng's error log shows
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for debian.example.com/debian/dists/bullseye-backports/InRelease -- check file permissions!
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye-backports/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for debian.example.com/debian/dists/bullseye-backports/InRelease -- check file permissions!
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for debian.example.com/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:23:15 2023|Failure to move file /var/cache/apt-cacher-ng/packages.microsoft.com/repos/code/dists/stable/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/packages.microsoft.com/repos/code/dists/stable/InRelease.1690870995 - errno: File exists
Tue Aug 1 08:23:15 2023|Error creating file item for packages.microsoft.com/repos/code/dists/stable/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/secdeb/dists/bullseye-security/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/secdeb/dists/bullseye-security/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for secdeb/dists/bullseye-security/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for download.virtualbox.org/virtualbox/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:23:45 2023|/var/cache/apt-cacher-ng/debrep/dists/bullseye/InRelease.1690871025 storage error [Checked size beyond EOF], check file AND directory permissions, last errno: File exists
Tue Aug 1 08:23:45 2023|Failure to move file /var/cache/apt-cacher-ng/dl.google.com/linux/chrome/deb/dists/stable/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/dl.google.com/linux/chrome/deb/dists/stable/InRelease.1690871025 - errno: File exists
Tue Aug 1 08:23:45 2023|Error creating file item for dl.google.com/linux/chrome/deb/dists/stable/InRelease -- check file permissions!
Tue Aug 1 08:24:16 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease.1690871056 - errno: File exists
Tue Aug 1 08:24:16 2023|Error creating file item for debian.example.com/debian/dists/bullseye/InRelease -- check file permissions!
Tue Aug 1 08:24:16 2023|Failure to move file /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/debian.example.com/debian/dists/bullseye/InRelease.1690871056 - errno: File exists
Tue Aug 1 08:24:16 2023|Error creating file item for debian.example.com/debian/dists/bullseye/InRelease -- check file permissions!
limits.conf is set to
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 16584
root hard nofile 16584
I cannot remember similar problems for Debian 10. Would it be possible
to make acng more reliable under heavy load?
Regards
Harri
0 new messages