Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1039710: debian-installer: Grub installation fails and /var/log/syslog is empty
50 views
Skip to first unread message
Roland Clobus
Jun 28, 2023, 9:10:04 AM6/28/23
to
Package: debian-installer
Version: daily build 2023-06-28T05:19Z
Severity: grave
Tags: d-i
Justification: renders package unusable
X-Debbugs-Cc: ph...@hands.com
Hello Debian-installer maintainers,
On openQA [1] the installation tests with the latest netinst image [2] fail,
because GRUB cannot install.
I've tried to look a bit deeper into the issue, but I cannot proceed, because
/var/log/syslog is empty. So effectively there are possibly two issues in this
report:
1) Failure in grub
2) No logging to /var/log/syslog
My findings so far:
* The command line arguments of syslogd and klogd (both from Busybox) have not
changed between Bookworm and Trixie.
* At the moment of the failure, the /var/log folder contains only 3 files [3]:
syslog (a single line, stating that syslog was started from Busybox [4]),
partman and Xorg.0.log
* When running `logger`, an entry should have been created in /var/log/syslog,
but that doesn't happen. The netinst image from Bookworm works correctly.
* Possibly relevant packages that have been updated: busybox, libc, linux-
image, bsdutils
With kind regards,
Roland Clobus
[1] https://openqa.debian.net/tests/167456
[2] https://get.debian.org/images/daily-builds/daily/arch-latest/amd64/iso-
cd/debian-testing-amd64-netinst.iso
[3] https://openqa.debian.net/tests/167456/file/grub-var_log.tar
[4] https://openqa.debian.net/tests/167456/logfile?filename=DI_syslog.txt
PS: Attached system information is from my personal computer, not the installed
system
-- System Information:
Debian Release: 12.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'testing-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 6.1.0-9-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Version: daily build 2023-06-28T05:19Z
Severity: grave
Tags: d-i
Justification: renders package unusable
X-Debbugs-Cc: ph...@hands.com
Hello Debian-installer maintainers,
On openQA [1] the installation tests with the latest netinst image [2] fail,
because GRUB cannot install.
I've tried to look a bit deeper into the issue, but I cannot proceed, because
/var/log/syslog is empty. So effectively there are possibly two issues in this
report:
1) Failure in grub
2) No logging to /var/log/syslog
My findings so far:
* The command line arguments of syslogd and klogd (both from Busybox) have not
changed between Bookworm and Trixie.
* At the moment of the failure, the /var/log folder contains only 3 files [3]:
syslog (a single line, stating that syslog was started from Busybox [4]),
partman and Xorg.0.log
* When running `logger`, an entry should have been created in /var/log/syslog,
but that doesn't happen. The netinst image from Bookworm works correctly.
* Possibly relevant packages that have been updated: busybox, libc, linux-
image, bsdutils
With kind regards,
Roland Clobus
[1] https://openqa.debian.net/tests/167456
[2] https://get.debian.org/images/daily-builds/daily/arch-latest/amd64/iso-
cd/debian-testing-amd64-netinst.iso
[3] https://openqa.debian.net/tests/167456/file/grub-var_log.tar
[4] https://openqa.debian.net/tests/167456/logfile?filename=DI_syslog.txt
PS: Attached system information is from my personal computer, not the installed
system
-- System Information:
Debian Release: 12.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'testing-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 6.1.0-9-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Cyril Brulebois
Jun 28, 2023, 5:01:35 PM6/28/23
to
Control: reassign -1 busybox-udeb 1:1.36.1-3
Cyril Brulebois <ki...@debian.org> (2023-06-28):
> busybox seems to me like the most likely suspect here. deb-reversion'ing
> bookworm's version as a version that's newer than the one in unstable,
> stashing its binaries under build/localudebs and building say a
> netboot(-gtk) image should be a quick way to test that hypothesis.
With a local build, confirmed -3 is buggy, and that reverting only
busybox-udeb to -1 is sufficient to restore syslog support in the
installer.
Reassigning there; the GRUB thing can be filed separately once we have
actual information via syslog.
Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Cyril Brulebois <ki...@debian.org> (2023-06-28):
> busybox seems to me like the most likely suspect here. deb-reversion'ing
> bookworm's version as a version that's newer than the one in unstable,
> stashing its binaries under build/localudebs and building say a
> netboot(-gtk) image should be a quick way to test that hypothesis.
With a local build, confirmed -3 is buggy, and that reverting only
busybox-udeb to -1 is sufficient to restore syslog support in the
installer.
Reassigning there; the GRUB thing can be filed separately once we have
actual information via syslog.
Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Roland Clobus
Jun 29, 2023, 11:31:46 AM6/29/23
to
Control: retitle -1 busybox-udeb: /var/log/syslog is empty
On 28/06/2023 22:54, Cyril Brulebois wrote:
> With a local build, confirmed -3 is buggy, and that reverting only
> busybox-udeb to -1 is sufficient to restore syslog support in the
> installer.
Confirmed and details to reproduce:
* Download the busybox binary file from [1] and extract the file `busybox`
* Run the latest netinst image in Qemu/KVM (sid)
* Select the installer
* Answer all the questions and let it run until an error (to make sure
that the network is properly configured)
* Select a shell in the installer
* Download the older busybox binary file (you can use my server)
`cd /`
`wget http://pioneers.game-host.org/busybox`
`chmod a+x busybox`
* Kill the running syslogd
`ps | grep syslogd`
`kill <the_pid>`
* Restart syslogd from the older busybox
`/busybox syslogd -m 0 -O /var/log/syslog -S`
* Log something
`logger -t Test It works now`
* Send Ctrl-Alt-F4, to see the output in the log
With kind regards,
Roland Clobus
[1]
https://snapshot.debian.org/archive/debian/20230608T144245Z/pool/main/b/busybox/busybox-udeb_1.36.1-1_amd64.udeb
On 28/06/2023 22:54, Cyril Brulebois wrote:
> With a local build, confirmed -3 is buggy, and that reverting only
> busybox-udeb to -1 is sufficient to restore syslog support in the
> installer.
* Download the busybox binary file from [1] and extract the file `busybox`
* Run the latest netinst image in Qemu/KVM (sid)
* Select the installer
* Answer all the questions and let it run until an error (to make sure
that the network is properly configured)
* Select a shell in the installer
* Download the older busybox binary file (you can use my server)
`cd /`
`wget http://pioneers.game-host.org/busybox`
`chmod a+x busybox`
* Kill the running syslogd
`ps | grep syslogd`
`kill <the_pid>`
* Restart syslogd from the older busybox
`/busybox syslogd -m 0 -O /var/log/syslog -S`
* Log something
`logger -t Test It works now`
* Send Ctrl-Alt-F4, to see the output in the log
With kind regards,
Roland Clobus
[1]
Philip Hands
Aug 7, 2023, 6:10:05 PM8/7/23
to
Steve McIntyre <st...@einval.com> writes:
> On Wed, Jul 12, 2023 at 11:15:57AM +0200, Cyril Brulebois wrote:
>>Hi Michael,
>>
>>Cyril Brulebois <ki...@debian.org> (2023-06-28):
>>have no logs for.
>
> After weeks with this breakage, I've just uploaded a minimal NMU to
> fix it, reverting the syslog changes since -1. I've buit and tested
> successfully locally.
Thanks -- and I agree, it works :-)
https://openqa.debian.net/tests/178534/logfile?filename=DI_syslog.txt
As it happens, over the weekend it occurred to me that I might be able
to pave the way to a fix for this bug by coming up with a test for the
failure.
Awkwardly, syslogd wants to open /dev/log and bails out if it's already
in use, so I resorted to (the somewhat disgusting hack of) using podman:
https://salsa.debian.org/philh/busybox/-/commit/2697f7cce81d1a70de202a30e7062dc9f64a94b1
At least it allows syslogd to run well enough to succeed or fail
similarly to the behaviour seen in the bug.
Here it is going wrong with the -3 code:
https://salsa.debian.org/philh/busybox/-/jobs/4523822#L3963
(lines 3969-3975, with the last line showing the entire syslog)
and here is an example of it going right:
https://salsa.debian.org/philh/busybox/-/jobs/4523808#L3668
Line 3668 here, saying "PASS: syslogd-works" indicates that we
succeeded in grepping the test string in /var/log/syslog
The difference between these two is simply disabling
CONFIG_FEATURE_REMOTE_LOG, as seen here:
https://salsa.debian.org/philh/busybox/-/commit/89c253f75690dd41487b6fd6f9356a1e890a6ac2
I'm not proposing that as a fix, but it does seem to indicate where the
problem may be located. I'm afraid I've failed to work out what's
actually going wrong here (my C's pretty rusty).
BTW At one point I thought I'd narrowed it down to the while loop here:
https://salsa.debian.org/philh/busybox/-/commit/328fdfbe43cd8d9e4425c3ee1c68aadfa44ee434
but if that did work, it does no longer. Either I was mistaken about it
having worked earlier (I'm at least 80% sure that's not the case) or
something non-deterministic is going on ... which makes me wonder if the
underlying cause might be something to do with using uninitialised data
somewhere.
Hopefully this will be of some use to those more familiar with the code.
Cheers, Phil.
--
Philip Hands -- https://hands.com/~phil
> On Wed, Jul 12, 2023 at 11:15:57AM +0200, Cyril Brulebois wrote:
>>Hi Michael,
>>
>>Cyril Brulebois <ki...@debian.org> (2023-06-28):
>>> Control: reassign -1 busybox-udeb 1:1.36.1-3
>>
>>[…]
>>
>>
>>> With a local build, confirmed -3 is buggy, and that reverting only
>>> busybox-udeb to -1 is sufficient to restore syslog support in the
>>> installer.
>>>
>>> Reassigning there; the GRUB thing can be filed separately once we have
>>> actual information via syslog.
>>
>>A fix would be appreciated, we've got reports piling up about things we
>>> With a local build, confirmed -3 is buggy, and that reverting only
>>> busybox-udeb to -1 is sufficient to restore syslog support in the
>>> installer.
>>>
>>> Reassigning there; the GRUB thing can be filed separately once we have
>>> actual information via syslog.
>>
>>have no logs for.
>
> After weeks with this breakage, I've just uploaded a minimal NMU to
> fix it, reverting the syslog changes since -1. I've buit and tested
> successfully locally.
Thanks -- and I agree, it works :-)
https://openqa.debian.net/tests/178534/logfile?filename=DI_syslog.txt
As it happens, over the weekend it occurred to me that I might be able
to pave the way to a fix for this bug by coming up with a test for the
failure.
Awkwardly, syslogd wants to open /dev/log and bails out if it's already
in use, so I resorted to (the somewhat disgusting hack of) using podman:
https://salsa.debian.org/philh/busybox/-/commit/2697f7cce81d1a70de202a30e7062dc9f64a94b1
At least it allows syslogd to run well enough to succeed or fail
similarly to the behaviour seen in the bug.
Here it is going wrong with the -3 code:
https://salsa.debian.org/philh/busybox/-/jobs/4523822#L3963
(lines 3969-3975, with the last line showing the entire syslog)
and here is an example of it going right:
https://salsa.debian.org/philh/busybox/-/jobs/4523808#L3668
Line 3668 here, saying "PASS: syslogd-works" indicates that we
succeeded in grepping the test string in /var/log/syslog
The difference between these two is simply disabling
CONFIG_FEATURE_REMOTE_LOG, as seen here:
https://salsa.debian.org/philh/busybox/-/commit/89c253f75690dd41487b6fd6f9356a1e890a6ac2
I'm not proposing that as a fix, but it does seem to indicate where the
problem may be located. I'm afraid I've failed to work out what's
actually going wrong here (my C's pretty rusty).
BTW At one point I thought I'd narrowed it down to the while loop here:
https://salsa.debian.org/philh/busybox/-/commit/328fdfbe43cd8d9e4425c3ee1c68aadfa44ee434
but if that did work, it does no longer. Either I was mistaken about it
having worked earlier (I'm at least 80% sure that's not the case) or
something non-deterministic is going on ... which makes me wonder if the
underlying cause might be something to do with using uninitialised data
somewhere.
Hopefully this will be of some use to those more familiar with the code.
Cheers, Phil.
--
Philip Hands -- https://hands.com/~phil
0 new messages