info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Performance degraded, Bookworm on Raspberry Pi 4B
3,533 views
Skip to first unread message
Hank Barta
Mar 14, 2022, 5:40:03 PM3/14/22
to
First of all, many many thanks for making Debian work on a Raspberry Pi 4B. I bounce back and forth between Bullseye and Bookworm on a Pi 4B and have noticed a performance degradation with Bookworm.
TL;DR
Install (current) Bookworm - performance good.
apt update/upgrade - performance tanks.
Revert 5.16 to 5.15 kernel - no improvement.
It seems that one of the 103 packages that get installed during the update causes this to happen.
More detail at https://github.com/HankB/Pi-4B-bookworm-performance
Is there anything I can do to help track this down? (I hope the answer is not "install the packages one at a time to see what causes this.")
Thanks!
--
Beautiful Sunny Winfield
Hank Barta
Mar 14, 2022, 10:40:03 PM3/14/22
to
I started installing packages I thought were likely to cause the regression (one or two at a time) and following installation of raspi-firmware the problem appeared.
Beautiful Sunny Winfield
Gunnar Wolf
Mar 15, 2022, 12:30:02 AM3/15/22
to
Hank Barta dijo [Mon, Mar 14, 2022 at 09:31:36PM -0500]:
Ouch.
raspi-firmware, as you know, is just a blob we cannot debug or do
anything about; it is a core component of our Raspberries, but... it
is non-free ☹
Given you have already pursued the situation to this detail, could you
share... between which versions do you see the regression?
FWIW, I cannot do much about it, other than report it as an issue in
https://github.com/raspberrypi/firmware/issues (which I will do) ;
taking a quick look at the reported issues, and found just the
following, although I don't know if they are in any way related to
your report:
https://github.com/raspberrypi/firmware/issues/1619
Greetings, and thanks for this report!
> I started installing packages I thought were likely to cause the regression
> (one or two at a time) and following installation of raspi-firmware the
> problem appeared.
As the maintainer of raspi-firmware, I have to say:
> (one or two at a time) and following installation of raspi-firmware the
> problem appeared.
Ouch.
raspi-firmware, as you know, is just a blob we cannot debug or do
anything about; it is a core component of our Raspberries, but... it
is non-free ☹
Given you have already pursued the situation to this detail, could you
share... between which versions do you see the regression?
FWIW, I cannot do much about it, other than report it as an issue in
https://github.com/raspberrypi/firmware/issues (which I will do) ;
taking a quick look at the reported issues, and found just the
following, although I don't know if they are in any way related to
your report:
https://github.com/raspberrypi/firmware/issues/1619
Greetings, and thanks for this report!
Diederik de Haas
Mar 15, 2022, 7:20:02 AM3/15/22
to
On Tuesday, 15 March 2022 05:20:02 CET Gunnar Wolf wrote:
> Given you have already pursued the situation to this detail, could you
> share... between which versions do you see the regression?
raspi-firmware/testing 1.20220120+ds-1 arm64 [upgradable from: 1.20210805+ds-1]> Given you have already pursued the situation to this detail, could you
> share... between which versions do you see the regression?
Hank Barta
Mar 15, 2022, 8:40:02 AM3/15/22
to
Some additional information that might be useful is that the system seems unable to determine CPU frequencies. For example
root@up:~# cpupower frequency-info
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
root@up:~# ls /sys/devices/system/cpu/cpu0/
cpu_capacity crash_notes_size node0/ power/ subsystem/ uevent
crash_notes hotplug/ of_node/ regs/ topology/
root@up:~# ls -l /sys/devices/system/cpu/cpu0/
total 0
-r--r--r-- 1 root root 4096 Mar 15 12:22 cpu_capacity
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes_size
drwxr-xr-x 2 root root 0 Mar 15 12:22 hotplug
lrwxrwxrwx 1 root root 0 Mar 15 12:22 node0 -> ../../node/node0
lrwxrwxrwx 1 root root 0 Mar 15 12:22 of_node -> ../../../../firmware/devicetree/base/cpus/cpu@0
drwxr-xr-x 2 root root 0 Mar 15 12:22 power
drwxr-xr-x 3 root root 0 Mar 15 12:22 regs
lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../bus/cpu
drwxr-xr-x 2 root root 0 Mar 15 12:16 topology
-rw-r--r-- 1 root root 4096 Jan 1 1970 uevent
root@up:~#
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
root@up:~# ls /sys/devices/system/cpu/cpu0/
cpu_capacity crash_notes_size node0/ power/ subsystem/ uevent
crash_notes hotplug/ of_node/ regs/ topology/
root@up:~# ls -l /sys/devices/system/cpu/cpu0/
total 0
-r--r--r-- 1 root root 4096 Mar 15 12:22 cpu_capacity
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes_size
drwxr-xr-x 2 root root 0 Mar 15 12:22 hotplug
lrwxrwxrwx 1 root root 0 Mar 15 12:22 node0 -> ../../node/node0
lrwxrwxrwx 1 root root 0 Mar 15 12:22 of_node -> ../../../../firmware/devicetree/base/cpus/cpu@0
drwxr-xr-x 2 root root 0 Mar 15 12:22 power
drwxr-xr-x 3 root root 0 Mar 15 12:22 regs
lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../bus/cpu
drwxr-xr-x 2 root root 0 Mar 15 12:16 topology
-rw-r--r-- 1 root root 4096 Jan 1 1970 uevent
root@up:~#
Whereas on a Pi 4B running Bullseye without this issue
hbarta@kweli:~$ cpupower frequency-info
analyzing CPU 0:
driver: cpufreq-dt
CPUs which run at the same hardware frequency: 0 1 2 3
CPUs which need to have their frequency coordinated by software: 0 1 2 3
maximum transition latency: Cannot determine or is not supported.
hardware limits: 600 MHz - 2.00 GHz
available frequency steps: 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz, 1.90 GHz, 2.00 GHz
available cpufreq governors: performance schedutil
current policy: frequency should be within 600 MHz and 2.00 GHz.
The governor "schedutil" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 600 MHz (asserted by call to kernel)
hbarta@kweli:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
2000000
hbarta@kweli:~$
analyzing CPU 0:
driver: cpufreq-dt
CPUs which run at the same hardware frequency: 0 1 2 3
CPUs which need to have their frequency coordinated by software: 0 1 2 3
maximum transition latency: Cannot determine or is not supported.
hardware limits: 600 MHz - 2.00 GHz
available frequency steps: 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz, 1.90 GHz, 2.00 GHz
available cpufreq governors: performance schedutil
current policy: frequency should be within 600 MHz and 2.00 GHz.
The governor "schedutil" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 600 MHz (asserted by call to kernel)
hbarta@kweli:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
2000000
hbarta@kweli:~$
The second example shows a Pi that has been overclocked and in fact, I discovered this issue when I tried to overclock a Pi 4B running Bookworm. I also discovered that the system did not respond to the settings in /boot/firmware/config.txt that are used to overclock. The following settings seem to have no effect
arm_freq=2000
over_voltage=6
gpu_freq=750
over_voltage=6
gpu_freq=750
best,
Beautiful Sunny Winfield
Diederik de Haas
Mar 15, 2022, 10:10:02 AM3/15/22
to
On Tuesday, 15 March 2022 13:32:37 CET Hank Barta wrote:
> Some additional information that might be useful is that the system seems
> unable to determine CPU frequencies. For example
>
> root@up:~# cpupower frequency-info
> analyzing CPU 0:
> no or unknown cpufreq driver is active on this CPU
> CPUs which run at the same hardware frequency: Not Available
> CPUs which need to have their frequency coordinated by software: Not
> Available
> maximum transition latency: Cannot determine or is not supported.
> Not Available
> available cpufreq governors: Not Available
> Unable to determine current policy
> current CPU frequency: Unable to call hardware
> current CPU frequency: Unable to call to kernel
I lack the knowledge to properly/fully understand the issue Gunnar linked to,
> Some additional information that might be useful is that the system seems
> unable to determine CPU frequencies. For example
>
> root@up:~# cpupower frequency-info
> analyzing CPU 0:
> no or unknown cpufreq driver is active on this CPU
> CPUs which run at the same hardware frequency: Not Available
> CPUs which need to have their frequency coordinated by software: Not
> Available
> maximum transition latency: Cannot determine or is not supported.
> Not Available
> available cpufreq governors: Not Available
> Unable to determine current policy
> current CPU frequency: Unable to call hardware
> current CPU frequency: Unable to call to kernel
but it looks applicable ... still (despite the last msg indicating the issue
was fixed).
I don't know exactly the role that clocks play, but they seem quite important.
And when the value of a/multiple clock(s) return 0, it doesn't surprise me
that it would cause other things to fail, like the ones you reported above,
which in turn cause the CPU (f.e.) to be way slower then it can be.
Personally I think it's more useful if Hank reported the issue himself to
https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.
Having a bug in Debian's BTS (against raspi-firmware) regarding this issue
seems useful as well and with the 'forwarded' keyword it can be linked to the
upstream issue.
Cheers,
Diederik
Hank Barta
Mar 15, 2022, 11:40:02 AM3/15/22
to
I tested this morning with an install on an SD card and the disk benchmarks were not significantly different. The updated `raspi-firmware` did seem to result in slightly slower performance (7188 events before and 6525 after) but I attribute this to a low CPU clock rather than a problem with the SD card driver. Details at https://github.com/HankB/Pi-4B-bookworm-performance#2022-03-15-testing-with-sd-card.
I'll be happy to file the bug reports. https://github.com/raspberrypi/firmware/issues/1705
Debian BTS next.
Beautiful Sunny Winfield
Gunnar Wolf
Mar 15, 2022, 12:10:02 PM3/15/22
to
Diederik de Haas dijo [Tue, Mar 15, 2022 at 03:01:59PM +0100]:
problems he is reporting than me, and it would be better if he were
able to send this information to the upstream authors; of course, it
makes sense to link to this thread.
Hank, are you OK with this suggestion?
> Personally I think it's more useful if Hank reported the issue himself to
> https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.
I agree with you, Diederik. I think Hank has a better grasp of the
> https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.
problems he is reporting than me, and it would be better if he were
able to send this information to the upstream authors; of course, it
makes sense to link to this thread.
Hank, are you OK with this suggestion?
Gunnar Wolf
Mar 15, 2022, 1:00:02 PM3/15/22
to
Package: raspi-firmware
Version: 1.20220120+ds-1
Severity: important
Tags: upstream
Hank Barta reported a performance regression under the Raspberry Pi
4B, after comparing Bullseye and Bookworm (Debian 11 and 12). I am
copying over to this bug report only part of the information; a
full(er) report is available in Hank's GitHub project
Pi-4B-bookworm-performance:
https://github.com/HankB/Pi-4B-bookworm-performance
The problem presented when running sysbench:
/--- bullseye ↓
| $ time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 584.72
|
| General statistics:
| total time: 10.0006s
| total number of events: 5850
| Latency (ms):
| min: 1.70
| avg: 1.71
| max: 2.99
| 95th percentile: 1.73
| sum: 9997.34
|
| Threads fairness:
| events (avg/stddev): 5850.0000/0.00
| execution time (avg/stddev): 9.9973/0.00
+--- bookworm ↓
| # time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 233.03
|
| General statistics:
| total time: 10.0038s
| total number of events: 2333
|
| Latency (ms):
| min: 4.28
| avg: 4.29
| max: 4.34
| 95th percentile: 4.33
| sum: 10001.67
|
| Threads fairness:
| events (avg/stddev): 2333.0000/0.00
| execution time (avg/stddev): 10.0017/0.00
\---
Hank did a thorough saerch, and found out this issue happens after the
firmware update (raspi-firmware/testing 1.20220120+ds-1 arm64
[upgradable from: 1.20210805+ds-1]).
Further information is available on the debian-arm mailing list
thread:
https://lists.debian.org/debian-arm/2022/03/threads.html#00003
I prompted Hank to report this issue to the Raspberrrypi Firmware
project in GitHub, which he did, but was outright dismissed:
https://github.com/raspberrypi/firmware/issues/1705
-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.16.0-3-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
--
Version: 1.20220120+ds-1
Severity: important
Tags: upstream
Hank Barta reported a performance regression under the Raspberry Pi
4B, after comparing Bullseye and Bookworm (Debian 11 and 12). I am
copying over to this bug report only part of the information; a
full(er) report is available in Hank's GitHub project
Pi-4B-bookworm-performance:
https://github.com/HankB/Pi-4B-bookworm-performance
The problem presented when running sysbench:
/--- bullseye ↓
| $ time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 584.72
|
| General statistics:
| total time: 10.0006s
| total number of events: 5850
| Latency (ms):
| min: 1.70
| avg: 1.71
| max: 2.99
| 95th percentile: 1.73
| sum: 9997.34
|
| Threads fairness:
| events (avg/stddev): 5850.0000/0.00
| execution time (avg/stddev): 9.9973/0.00
+--- bookworm ↓
| # time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 233.03
|
| General statistics:
| total time: 10.0038s
| total number of events: 2333
|
| Latency (ms):
| min: 4.28
| avg: 4.29
| max: 4.34
| 95th percentile: 4.33
| sum: 10001.67
|
| Threads fairness:
| events (avg/stddev): 2333.0000/0.00
| execution time (avg/stddev): 10.0017/0.00
\---
Hank did a thorough saerch, and found out this issue happens after the
firmware update (raspi-firmware/testing 1.20220120+ds-1 arm64
[upgradable from: 1.20210805+ds-1]).
Further information is available on the debian-arm mailing list
thread:
https://lists.debian.org/debian-arm/2022/03/threads.html#00003
I prompted Hank to report this issue to the Raspberrrypi Firmware
project in GitHub, which he did, but was outright dismissed:
https://github.com/raspberrypi/firmware/issues/1705
-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.16.0-3-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
--
0 new messages