Effective February 22, 2024, Google Groups will no longer support new Usenet content. Posting and subscribing will be disallowed, and new content from Usenet peers will not appear. Viewing and searching of historical data will still be supported as it is done today.
Dismiss

Performance degraded, Bookworm on Raspberry Pi 4B

2,777 views
Skip to first unread message

Hank Barta

unread,
Mar 14, 2022, 5:40:03 PM3/14/22
to
First of all, many many thanks for making Debian work on a Raspberry Pi 4B. I bounce back and forth between  Bullseye and Bookworm on a Pi 4B and have noticed a performance degradation with Bookworm.

TL;DR
Install (current) Bookworm - performance good.
apt update/upgrade - performance tanks.
Revert 5.16 to 5.15 kernel - no improvement.

It seems that one of the 103 packages that get installed during the update causes this to happen.


Is there anything I can do to help track this down? (I hope the answer is not "install the packages one at a time to see what causes this.")

Thanks!
--
Beautiful Sunny Winfield

Hank Barta

unread,
Mar 14, 2022, 10:40:03 PM3/14/22
to
I started installing packages I thought were likely to cause the regression (one or two at a time) and following installation of raspi-firmware the problem appeared. 
--
Beautiful Sunny Winfield

Gunnar Wolf

unread,
Mar 15, 2022, 12:30:02 AM3/15/22
to
Hank Barta dijo [Mon, Mar 14, 2022 at 09:31:36PM -0500]:
> I started installing packages I thought were likely to cause the regression
> (one or two at a time) and following installation of raspi-firmware the
> problem appeared.

As the maintainer of raspi-firmware, I have to say:

Ouch.


raspi-firmware, as you know, is just a blob we cannot debug or do
anything about; it is a core component of our Raspberries, but... it
is non-free ☹

Given you have already pursued the situation to this detail, could you
share... between which versions do you see the regression?

FWIW, I cannot do much about it, other than report it as an issue in
https://github.com/raspberrypi/firmware/issues (which I will do) ;
taking a quick look at the reported issues, and found just the
following, although I don't know if they are in any way related to
your report:

https://github.com/raspberrypi/firmware/issues/1619

Greetings, and thanks for this report!

Diederik de Haas

unread,
Mar 15, 2022, 7:20:02 AM3/15/22
to
On Tuesday, 15 March 2022 05:20:02 CET Gunnar Wolf wrote:
> Given you have already pursued the situation to this detail, could you
> share... between which versions do you see the regression?

raspi-firmware/testing 1.20220120+ds-1 arm64 [upgradable from: 1.20210805+ds-1]
signature.asc

Hank Barta

unread,
Mar 15, 2022, 8:40:02 AM3/15/22
to
Some additional information that might be useful is that the system seems unable to determine CPU frequencies. For example

root@up:~# cpupower frequency-info
analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  CPUs which run at the same hardware frequency: Not Available
  CPUs which need to have their frequency coordinated by software: Not Available
  maximum transition latency:  Cannot determine or is not supported.
Not Available
  available cpufreq governors: Not Available
  Unable to determine current policy
  current CPU frequency: Unable to call hardware
  current CPU frequency:  Unable to call to kernel
root@up:~# ls /sys/devices/system/cpu/cpu0/      
cpu_capacity      crash_notes_size  node0/            power/            subsystem/        uevent            
crash_notes       hotplug/          of_node/          regs/             topology/        
root@up:~# ls -l /sys/devices/system/cpu/cpu0/
total 0
-r--r--r-- 1 root root 4096 Mar 15 12:22 cpu_capacity
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes
-r-------- 1 root root 4096 Mar 15 12:22 crash_notes_size
drwxr-xr-x 2 root root    0 Mar 15 12:22 hotplug
lrwxrwxrwx 1 root root    0 Mar 15 12:22 node0 -> ../../node/node0
lrwxrwxrwx 1 root root    0 Mar 15 12:22 of_node -> ../../../../firmware/devicetree/base/cpus/cpu@0
drwxr-xr-x 2 root root    0 Mar 15 12:22 power
drwxr-xr-x 3 root root    0 Mar 15 12:22 regs
lrwxrwxrwx 1 root root    0 Jan  1  1970 subsystem -> ../../../../bus/cpu
drwxr-xr-x 2 root root    0 Mar 15 12:16 topology
-rw-r--r-- 1 root root 4096 Jan  1  1970 uevent
root@up:~#

Whereas on a Pi 4B running Bullseye without this issue

hbarta@kweli:~$ cpupower frequency-info
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 600 MHz - 2.00 GHz
  available frequency steps:  600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz, 1.90 GHz, 2.00 GHz
  available cpufreq governors: performance schedutil
  current policy: frequency should be within 600 MHz and 2.00 GHz.
                  The governor "schedutil" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 600 MHz (asserted by call to kernel)
hbarta@kweli:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
2000000
hbarta@kweli:~$
 

The second example shows a Pi that has been overclocked and in fact, I discovered this issue when I tried to overclock a Pi 4B running Bookworm. I also discovered that the system did not respond to the settings in /boot/firmware/config.txt that are used to overclock. The following settings seem to have no effect

arm_freq=2000
over_voltage=6
gpu_freq=750


 best,
--
Beautiful Sunny Winfield

Diederik de Haas

unread,
Mar 15, 2022, 10:10:02 AM3/15/22
to
On Tuesday, 15 March 2022 13:32:37 CET Hank Barta wrote:
> Some additional information that might be useful is that the system seems
> unable to determine CPU frequencies. For example
>
> root@up:~# cpupower frequency-info
> analyzing CPU 0:
> no or unknown cpufreq driver is active on this CPU
> CPUs which run at the same hardware frequency: Not Available
> CPUs which need to have their frequency coordinated by software: Not
> Available
> maximum transition latency: Cannot determine or is not supported.
> Not Available
> available cpufreq governors: Not Available
> Unable to determine current policy
> current CPU frequency: Unable to call hardware
> current CPU frequency: Unable to call to kernel

I lack the knowledge to properly/fully understand the issue Gunnar linked to,
but it looks applicable ... still (despite the last msg indicating the issue
was fixed).
I don't know exactly the role that clocks play, but they seem quite important.
And when the value of a/multiple clock(s) return 0, it doesn't surprise me
that it would cause other things to fail, like the ones you reported above,
which in turn cause the CPU (f.e.) to be way slower then it can be.

Personally I think it's more useful if Hank reported the issue himself to
https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.

Having a bug in Debian's BTS (against raspi-firmware) regarding this issue
seems useful as well and with the 'forwarded' keyword it can be linked to the
upstream issue.

Cheers,
Diederik
signature.asc

Hank Barta

unread,
Mar 15, 2022, 11:40:02 AM3/15/22
to
I tested this morning with an install on an SD card and the disk benchmarks were not significantly different. The updated `raspi-firmware` did seem to result in slightly slower performance (7188 events before and 6525 after) but I attribute this to a low CPU clock rather than a problem with the SD card driver. Details at https://github.com/HankB/Pi-4B-bookworm-performance#2022-03-15-testing-with-sd-card.

I'll be happy to file the  bug reports. https://github.com/raspberrypi/firmware/issues/1705

Debian BTS next.
--
Beautiful Sunny Winfield

Gunnar Wolf

unread,
Mar 15, 2022, 12:10:02 PM3/15/22
to
Diederik de Haas dijo [Tue, Mar 15, 2022 at 03:01:59PM +0100]:
> Personally I think it's more useful if Hank reported the issue himself to
> https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.

I agree with you, Diederik. I think Hank has a better grasp of the
problems he is reporting than me, and it would be better if he were
able to send this information to the upstream authors; of course, it
makes sense to link to this thread.

Hank, are you OK with this suggestion?
signature.asc

Gunnar Wolf

unread,
Mar 15, 2022, 1:00:02 PM3/15/22
to
Package: raspi-firmware
Version: 1.20220120+ds-1
Severity: important
Tags: upstream

Hank Barta reported a performance regression under the Raspberry Pi
4B, after comparing Bullseye and Bookworm (Debian 11 and 12). I am
copying over to this bug report only part of the information; a
full(er) report is available in Hank's GitHub project
Pi-4B-bookworm-performance:

https://github.com/HankB/Pi-4B-bookworm-performance

The problem presented when running sysbench:

/--- bullseye ↓
| $ time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 584.72
|
| General statistics:
| total time: 10.0006s
| total number of events: 5850
| Latency (ms):
| min: 1.70
| avg: 1.71
| max: 2.99
| 95th percentile: 1.73
| sum: 9997.34
|
| Threads fairness:
| events (avg/stddev): 5850.0000/0.00
| execution time (avg/stddev): 9.9973/0.00
+--- bookworm ↓
| # time sysbench --test=cpu --cpu-max-prime=20000 run
| (...)
| CPU speed:
| events per second: 233.03
|
| General statistics:
| total time: 10.0038s
| total number of events: 2333
|
| Latency (ms):
| min: 4.28
| avg: 4.29
| max: 4.34
| 95th percentile: 4.33
| sum: 10001.67
|
| Threads fairness:
| events (avg/stddev): 2333.0000/0.00
| execution time (avg/stddev): 10.0017/0.00
\---

Hank did a thorough saerch, and found out this issue happens after the
firmware update (raspi-firmware/testing 1.20220120+ds-1 arm64
[upgradable from: 1.20210805+ds-1]).

Further information is available on the debian-arm mailing list
thread:

https://lists.debian.org/debian-arm/2022/03/threads.html#00003

I prompted Hank to report this issue to the Raspberrrypi Firmware
project in GitHub, which he did, but was outright dismissed:

https://github.com/raspberrypi/firmware/issues/1705

-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.16.0-3-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

--

signature.asc
Reply all
Reply to author
Forward
0 new messages