hangs at boot

[jd]

Hi!

I just installed debian 11, it keeps hanging at boot at the same place.

It gets to

[ OK ] Started Make remote CUPS printers available locally

and then it hangs. I don't know where I should even begin to look in
order to diagnose this, anyone got any ideas?


cheers

jd

[Hans]

Am Mittwoch, 30. November 2022, 18:33:49 CET schrieb jd:

Try to deinstall cups and remove any usb cameras. I had had the problem, that
cups detected an usb camera as a printer and then stopped booting.

The camera was a noname one, and I do not own it any more.

Just an idea....

Good luck!

Hans

[Greg Wooledge]

On Thu, Dec 01, 2022 at 01:36:05AM +0100, jd wrote:

>
> On 2022-11-30 18:52, Hans wrote:
> > Am Mittwoch, 30. November 2022, 18:33:49 CET schrieb jd:
> >
> > Try to deinstall cups and remove any usb cameras. I had had the problem, that
> > cups detected an usb camera as a printer and then stopped booting.
>
>

> I tried removing all cups packages, then it just stalled at Unattended
> upgrades shutdown instead.
>
> So I tried disabling that, and now it stalls at Hostname Service.
>
> I'm guessing there's some underlying issue that hasn't got anything to do
> with these services but I have no idea what it might be.

Have you considered the possibility it's *not* hung, but is in fact
booted and working?

Have you tried pressing Enter a few times to see if a new login prompt
gets printed?

[jd]

On 2022-11-30 18:52, Hans wrote:

> Am Mittwoch, 30. November 2022, 18:33:49 CET schrieb jd:
>
> Try to deinstall cups and remove any usb cameras. I had had the problem, that
> cups detected an usb camera as a printer and then stopped booting.

I tried removing all cups packages, then it just stalled at Unattended
upgrades shutdown instead.

So I tried disabling that, and now it stalls at Hostname Service.

I'm guessing there's some underlying issue that hasn't got anything to
do with these services but I have no idea what it might be.

>

> The camera was a noname one, and I do not own it any more.
>
> Just an idea....
>
> Good luck!

Thanks

cheers

jd

[David Wright]

Quite often I lose my Login: or Password: prompt, caused by something
having the effect of sending Ctrl-A Ctrl-K to the console—a minor
inconvenience that I've never bothered to investigate.
But I've never managed to lose both the /etc/issue output and the
Login: line.

If pressing Enter doesn't work, it might be that the "main" VC is
broken; perhaps try switching to another VC with Alt-→ and seeing
if a Login: prompt appears there.

OTOH perhaps the /sbin/agetty is completely screwed up.

Cheers,
David.

[David Wright]

You could try booting into single-user mode, which shouldn't start
many of the services. Then check out the logs from the previous boots
to see what happened.

Cheers,
David.

[Tixy]

On Thu, 2022-12-01 at 01:36 +0100, jd wrote:
> On 2022-11-30 18:52, Hans wrote:
> > Am Mittwoch, 30. November 2022, 18:33:49 CET schrieb jd:
> >
> > Try to deinstall cups and remove any usb cameras. I had had the problem, that
> > cups detected an usb camera as a printer and then stopped booting.
>
>
> I tried removing all cups packages, then it just stalled at Unattended
> upgrades shutdown instead.
>
> So I tried disabling that, and now it stalls at Hostname Service.
>
> I'm guessing there's some underlying issue that hasn't got anything to
> do with these services but I have no idea what it might be.

The above, and presumably 'remote CUPS printers' in the earlier post,
involve the network and related services like DNS or DHCP.

-- 
Tixy

[jd]

Pressing enter doesn't do anything, but I can switch VC. I get a login
prompt on every VC but the first one. The network is up to, I get an
ip-address and I can ping the internet. Neither sddm or kde starts
though, nor does my home partition get mounted. In fact it doesn't even
show up in /etc/fstab and I definitely configured it to get mounted on
nvme0n1p1 during installation.


Perhaps there's just something iffy with the installation media I used?

[Thomas Schmitt]

Hi,

jd wrote:
> Perhaps there's just something iffy with the installation media I used?

I deem this unlikely, given that you report no errors during installation.

Nevertheless:

Which installation image did you use ?

If the installation medium with that image is still at hand, what checksum
do you get from it ?

You will have to curb the checksumming to the size of the image.
E.g.:

$ ls -l debian-11.5.0-amd64-netinst.iso
-rw-r--r-- 1 thomas thomas 400556032 Sep 10 14:40 debian-11.5.0-amd64-netinst.iso
$ expr 400556032 / 2048
195584
$ dd if=/dev/sdd bs=2048 count=195584 | sha512sum
195584+0 records in
195584+0 records out
400556032 bytes (401 MB) copied, 2.54246 s, 158 MB/s
6a6607a05d57b7c62558e9c462fe5c6c04b9cfad2ce160c3e9140aa4617ab73aff7f5f745dfe51bbbe7b33c9b0e219a022ad682d6c327de0e53e40f079abf66a -

Then read the image's official SHA512 from its SHA512SUMS file. With above
ISO this is at
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
Compare it with the result of above run:

$ test 6a6607a05d57b7c62558e9c462fe5c6c04b9cfad2ce160c3e9140aa4617ab73aff7f5f745dfe51bbbe7b33c9b0e219a022ad682d6c327de0e53e40f079abf66a = 6a6607a05d57b7c62558e9c462fe5c6c04b9cfad2ce160c3e9140aa4617ab73aff7f5f745dfe51bbbe7b33c9b0e219a022ad682d6c327de0e53e40f079abf66a && echo MATCH
MATCH


Have a nice day :)

Thomas

[David Wright]

On Thu 01 Dec 2022 at 13:22:11 (+0100), jd wrote:
> On 2022-12-01 04:14, David Wright wrote:
> > On Wed 30 Nov 2022 at 19:38:05 (-0500), Greg Wooledge wrote:

> > > Have you considered the possibility it's *not* hung, but is in fact
> > > booted and working?
> > >
> > > Have you tried pressing Enter a few times to see if a new login prompt
> > > gets printed?
> >

> > If pressing Enter doesn't work, it might be that the "main" VC is
> > broken; perhaps try switching to another VC with Alt-→ and seeing
> > if a Login: prompt appears there.
>
> Pressing enter doesn't do anything, but I can switch VC. I get a login
> prompt on every VC but the first one. The network is up to, I get an
> ip-address and I can ping the internet. Neither sddm or kde starts
> though, nor does my home partition get mounted. In fact it doesn't
> even show up in /etc/fstab and I definitely configured it to get
> mounted on nvme0n1p1 during installation.

Presumably you logged in as root? Is there a /home directory at all?
If there is, does it contain anything, like a directory with two or
three dotfiles in it?

What are the contents of /etc/fstab?

Although /var/log/installer/partman contains the general layout of the
disk partitions (right near the end), it doesn't reflect the final
screen that you see in the d-i's partitioner. Do you have a record
of the latter? It might look something like this:

┌────────────────────────┤ [!!] Partition disks ├─────────────────────────┐
│ │
│ This is an overview of your currently configured partitions and mount │
│ points. Select a partition to modify its settings (file system, mount │
│ point, etc.), a free space to create partitions, or a device to │
│ initialize its partition table. │
│ │
│ Guided partitioning │
│ Configure software RAID │
│ Configure the Logical Volume Manager │
│ Configure encrypted volumes │
│ Configure iSCSI volumes │
│ │
│ SCSI1 (0,0,0) (sda) - 500.1 GB ATA WDC WD5000ABCD-0 │
│ > #1 primary 30.9 GB B F ext4 / │
│ > #2 primary 31.1 GB ext4 │
│ > #3 primary 434.1 GB K ext4 /home │
│ > #4 primary 4.0 GB swap │
│ SCSI5 (0,0,0) (sdc) - 2.0 GB Generic Flash Disk │
│ │
│ Undo changes to partitions │
│ Finish partitioning and write changes to disk ← selected │
│ │
│ <Go Back> │
│ │
└─────────────────────────────────────────────────────────────────────────┘

(In the case of this example, /home was being inherited from a previous
installation in the 2nd partition, hence the K(eep) rather than F(ormat).)

Close to the end of /var/log/installer/syslog, there should be a
section like:

$ grep -B1 -A10 -h 'Adding new group' /var/log/installer/syslog
Apr 17 01:42:21 user-setup: Adding user `auser' ...
Apr 17 01:42:21 user-setup: Adding new group `auser' (1000) ...
Apr 16 20:42:21 groupadd[29019]: group added to /etc/group: name=auser, GID=1000
Apr 16 20:42:21 groupadd[29019]: group added to /etc/gshadow: name=auser
Apr 16 20:42:21 groupadd[29019]: new group: name=auser, GID=1000
Apr 17 01:42:21 user-setup: Adding new user `auser' (1000) with group `auser' ...
Apr 16 20:42:21 useradd[29025]: new user: name=auser, UID=1000, GID=1000, home=/home/auser, shell=/bin/bash, from=none
Apr 17 01:42:21 user-setup: Creating home directory `/home/auser' ...
Apr 17 01:42:21 user-setup: Copying files from `/etc/skel' ...
Apr 16 20:42:21 usermod[29034]: change user 'auser' password
Apr 16 20:42:21 chfn[29041]: changed user 'auser' information
Apr 16 20:42:21 chpasswd[29050]: pam_unix(chpasswd:chauthtok): password changed for auser
$

If you don't have that, it may be that the installation didn't
complete correctly. Although you type the username and password near
the start of the installation, the d-i doesn't actually write it
on the disk until moments before you're asked to remove the
installation media.

Cheers,
David.

[Tim Woodall]

On Thu, 1 Dec 2022, jd wrote:

>
> On 2022-11-30 18:52, Hans wrote:
>> Am Mittwoch, 30. November 2022, 18:33:49 CET schrieb jd:
>>
>> Try to deinstall cups and remove any usb cameras. I had had the problem,
>> that
>> cups detected an usb camera as a printer and then stopped booting.
>
>
> I tried removing all cups packages, then it just stalled at Unattended
> upgrades shutdown instead.
>
> So I tried disabling that, and now it stalls at Hostname Service.
>

is it stalled or just taking a very long time? DNS timing out can
sometimes cause issues like this.

[jd]

On 2022-12-01 18:19, David Wright wrote:
> On Thu 01 Dec 2022 at 13:22:11 (+0100), jd wrote:
>> On 2022-12-01 04:14, David Wright wrote:
>>> On Wed 30 Nov 2022 at 19:38:05 (-0500), Greg Wooledge wrote:
>>>> Have you considered the possibility it's *not* hung, but is in fact
>>>> booted and working?
>>>>
>>>> Have you tried pressing Enter a few times to see if a new login prompt
>>>> gets printed?
>>> If pressing Enter doesn't work, it might be that the "main" VC is
>>> broken; perhaps try switching to another VC with Alt-→ and seeing
>>> if a Login: prompt appears there.
>> Pressing enter doesn't do anything, but I can switch VC. I get a login
>> prompt on every VC but the first one. The network is up to, I get an
>> ip-address and I can ping the internet. Neither sddm or kde starts
>> though, nor does my home partition get mounted. In fact it doesn't
>> even show up in /etc/fstab and I definitely configured it to get
>> mounted on nvme0n1p1 during installation.
> Presumably you logged in as root? Is there a /home directory at all?
> If there is, does it contain anything, like a directory with two or
> three dotfiles in it?

I've reinstalled again after checking the checksum for the iso file was
correct and after changing to a new usb-stick to make sure that wasn't
the problem. I now have /home mounted on nvme0n1p1 where it is supposed
to be, and it shows up in fstab. In spite of what I said earlier I think
I simply forgot to configure the home partition on the previous
installation. It has all the normal dotfiles, . , .. , .bash_logout ,
.bashrc , .config and .profile


The problem still persists though.

I should also mention that I can log in as a regular user.


cheers

jd

[Greg Wooledge]

On Thu, Dec 01, 2022 at 07:25:24PM +0100, jd wrote:

> That depends on how long it can get stuck due to DNS issues, I've left it
> for several hours and it's still stuck in the same place. If it was a DNS
> issue it should have gotten somewhere after several hours no?

But when you say "stuck", what you really mean is "I never get a login
prompt on tty1". But you *are* getting login prompts on other TTYs,
right?

Which installation image did you use, and how deeply have you investigated
the systemd getty/console startup stuff?

[jd]

On 2022-12-01 19:05, Tim Woodall wrote:

[email....@gmail.com]

Yes exactly.

Which installation image did you use, 

https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.5.0-amd64-netinst.iso

and how deeply have you investigated
the systemd getty/console startup stuff?

Not very, I wouldn't really know how.

As it turns out I didn't need to but for future use, do you have any tips on where one might find documentation about how agetty interacts with the rest of the debian startup process? I've searched but the results have been scant.

The problem is now solved, but unsatisfactorily I still don't know what the problem was or exactly what solved it.

Anyhow what I did was to reinstall in bios-mode instead of uefi mode. After that the problem remained but to my surprise startx now worked, which it didn't before. After that I installed a bunch of packages that I usually do after a fresh install, one of which was lightdm, and all of a sudden everything works and the problem is gone.

Thanks to everyone for all the help and suggestions along the way.

cheers

jd  

[Greg Wooledge]

On Fri, Dec 09, 2022 at 01:19:46AM +0100, email....@gmail.com wrote:
> As it turns out I didn't need to but for future use, do you have any tips on
> where one might find documentation about how agetty interacts with the rest
> of the debian startup process? I've searched but the results have been
> scant.

I don't know all of the pieces either. There are a LOT of them.

By default, systemd is supposed to run a single getty on /dev/tty1, and
also passively "listen" for activity on tty2 through tty6. Switching
to one of those VTs will cause a getty to be spawned there. NOT switching
to them leaves them with nothing visibly running.

Before pressing Ctrl-Alt-F3:

unicorn:~$ ps -ft tty3
UID PID PPID C STIME TTY TIME CMD

After pressing Ctrl-Alt-F3:

unicorn:~$ ps -ft tty3
UID PID PPID C STIME TTY TIME CMD
root 852799 1 0 20:18 tty3 00:00:00 /sbin/agetty -o -p -- \u --n

The number of passively launched getty processes is configurable in
/etc/systemd/logind.conf. I believe it's the NAutoVTs= parameter that
controls this.

I've forgotten the details of your problem, but if switching to tty2
gives you a login prompt, but switching back to tty1 does NOT give you
one on tty1, then there must be something going wrong with the getty
that's supposed to be started on tty1 at boot time -- but the passive
gettys controlled by systemd-logind must be working.

I think the passive ones are controlled by getty@.service (the @ sign is
some kind of wildcard). If I read /lib/systemd/system/getty@.service
I can see

ExecStart=-/sbin/agetty -o '-p -- \\u' --noclear %I $TERM

which appears to match the process I'm seeing on tty3:

unicorn:~$ ps w -ft tty3
UID PID PPID C STIME TTY STAT TIME CMD
root 852799 1 0 20:18 tty3 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear tty3 linux

I don't know whether the initial getty on tty1 is also controlled by
this service, or by some other service. (I've already logged in on tty1
and done a "startx", so my whole X session is running on tty1, instead
of a getty.) For grins:

unicorn:~$ systemctl status ge...@tty1.service
● ge...@tty1.service - Getty on tty1
Loaded: loaded (/lib/systemd/system/getty@.service; enabled; vendor preset>
Drop-In: /etc/systemd/system/getty@.service.d
└─noclear.conf
Active: active (running) since Thu 2022-11-17 18:05:49 EST; 3 weeks 0 days>
Docs: man:agetty(8)
man:systemd-getty-generator(8)
http://0pointer.de/blog/projects/serial-console.html
Main PID: 743 (login)
Tasks: 0 (limit: 14198)
Memory: 1.7M
CPU: 48ms
CGroup: /system.slice/system-getty.slice/ge...@tty1.service
‣ 743 /bin/login -p --

Warning: some journal files were not opened due to insufficient permissions.

Ah, right, I configured that noclear.conf file on this system.

unicorn:~$ cat /etc/systemd/system/getty@.service.d/noclear.conf
[Service]
TTYVTDisallocate=no

So I guess getty@.service also controls the getty on tty1. I have no idea
why it would stop working. Check your logs (systemctl/journalctl as root)
and see what you can find.

Even though I'm still not clear on how all this stuff works, I hope some
of this rambling and pasting might be helpful.

[David Wright]

I've removed [Solved] from the Subject: line as reinstallation
doesn't count as a solution, and any evidence is destroyed.

$ tail -n 3 /lib/systemd/system/getty@.service
[Install]
WantedBy=getty.target
DefaultInstance=tty1
$

This has the effect of installing the symlink
/etc/systemd/system/getty.target.wants/ge...@tty1.service
pointing to /lib/systemd/system/getty@.service, which
isn't used at runtime, but sets up the symlink that actually
starts the tty1 getty. AIUI it also prevents any /automatic/
(passive) instantiation on tty1.

(If you remember back to September, you can prevent the appearance
of the tty1 getty with systemctl disable ge...@tty1.service,
which removes the symlink above. See:

https://lists.debian.org/debian-user/2022/09/msg00661.html

You and I presumably run our systems with multi-user as the final
target. I think the OP had graphical instead, but hadn't installed
a DM. That's how I interpret:

"After that I installed a bunch of packages that I usually do after
a fresh install, one of which was lightdm, and all of a sudden
everything works and the problem is gone."

which implies they hadn't done this first time around.

Somebody might test this on a DM/DE system by removing the DM.
I imagine that tty1 will sit there waiting for the DM to appear.

Obviously I don't know how one would manage to get this with the d-i,
but as the OP appears to have had UEFI/BIOS issues, that might mean
that the d-i failed in a manner we haven't been told about.

Cheers,
David.

[Greg Wooledge]

On Fri, Dec 09, 2022 at 01:57:44PM -0600, David Wright wrote:
> You and I presumably run our systems with multi-user as the final
> target. I think the OP had graphical instead, but hadn't installed
> a DM.

I didn't change the default target.

unicorn:~$ systemctl get-default
graphical.target

In the absence of a DM package of any kind, it gives me a getty/login
prompt on tty1. It's never failed yet, so I've never had to dive deeply
into how it actually works.

> That's how I interpret:
>
> "After that I installed a bunch of packages that I usually do after
> a fresh install, one of which was lightdm, and all of a sudden
> everything works and the problem is gone."
>
> which implies they hadn't done this first time around.
>
> Somebody might test this on a DM/DE system by removing the DM.
> I imagine that tty1 will sit there waiting for the DM to appear.

I have no idea what would happen if one were to install a DM package,
reboot, and then remove it. I'm not brave/curious/bored enough to
try it myself, either.

> Obviously I don't know how one would manage to get this with the d-i,
> but as the OP appears to have had UEFI/BIOS issues, that might mean
> that the d-i failed in a manner we haven't been told about.

Yeah. And if the whole system has been wiped and reinstalled, I doubt
we'll ever learn the answer.

[David Wright]

On Fri 09 Dec 2022 at 15:22:23 (-0500), Greg Wooledge wrote:
> On Fri, Dec 09, 2022 at 01:57:44PM -0600, David Wright wrote:

> > You and I presumably run our systems with multi-user as the final^H^H^H^H^H
default

> > target. I think the OP had graphical instead, but hadn't installed
> > a DM.
>
> I didn't change the default target.
>
> unicorn:~$ systemctl get-default
> graphical.target
>
> In the absence of a DM package of any kind, it gives me a getty/login
> prompt on tty1. It's never failed yet, so I've never had to dive deeply
> into how it actually works.

You're right. I does appear that the postinst script of the DM
is what creates the symlink in /etc/to trigger the DM.

> > but as the OP appears to have had UEFI/BIOS issues, that might mean
> > that the d-i failed in a manner we haven't been told about.

… and I'd forgotten about the small matter of their lacking the home partition.

> Yeah. And if the whole system has been wiped and reinstalled, I doubt
> we'll ever learn the answer.

Agreed.

Cheers,
David.