Upgrade to Bookworm, now GNOME keyring dies--no access to stored SSH key passwords

[Nate Bargmann]

I now have two desktop systems running Bookworm with GNOME. The laptop
was upgraded last month and I upgraded the desktop this afternoon. I
have been using the GNOME keyring applet to manage the SSH public key
passwords I use as it prompts to save passwords and then lets me SSH to
other hosts without out a password prompt.

Some time after the upgrade I wanted to SSH into one of the other
systems on my LAN and was greeted with a password prompt for the
corresponding public key that had prior been managed by the keyring
applet. I noted differences in the running processes between the laptop
where the keyring applet is still working and the desktop where it was
not.

On an off-chance I cold booted this system and found the keyring applet
was working as expected so I went on doing other things for a while.
Then I tried again and was prompted for the public key's password.
Uggh.

Right after rebooting the process list looked like this which mirrors
the laptop:

$ ps ax -u nate | grep "agent\|keyring"
2037 ? SLsl 0:00 /usr/bin/gnome-keyring-daemon --foreground --components=pkcs11,secrets --control-directory=/run/user/1000/keyring
2151 ? Ssl 0:00 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr
2157 ? Ss 0:00 ssh-agent -D -a /run/user/1000/openssh_agent
3802 ? S 0:00 /usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh
3922 pts/0 S+ 0:00 grep --color=auto agent\|keyring

When I began this mail things looked like this:

$ ps ax -u nate | grep "agent\|keyring"
2151 ? Ssl 0:00 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr
2157 ? Ss 0:00 ssh-agent -D -a /run/user/1000/openssh_agent
12324 ? Sl 0:00 /usr/bin/gnome-keyring-daemon --start --foreground --components=secrets
12325 ? Ssl 0:00 /usr/bin/gnome-keyring-daemon --foreground --components=pkcs11,secrets --control-directory=/run/user/1000/keyring
19308 pts/0 S+ 0:00 grep --color=auto agent\|keyring

It appears to me that gnome-keyring-daemon has been restarted for some reason.
As a result PIDs 2037 and 3802 are terminated and also
/run/user/1000/keyring/.ssh is no longer present along with the pkcs11 and ssh
files in the same directory.

I don't see anything out of the ordinary, in fact, these packages are
the same on the desktop and laptop systems:

debian-archive-keyring/stable,stable,now 2023.3 all [installed,automatic]
fasttrack-archive-keyring/stable,stable,now 2020.12.19 all [installed]
gnome-keyring-pkcs11/stable,now 42.1-1+b2 amd64 [installed,automatic]
gnome-keyring/stable,now 42.1-1+b2 amd64 [installed,automatic]
gpg-agent/stable,now 2.2.40-1.1 amd64 [installed,automatic]
libpam-gnome-keyring/stable,now 42.1-1+b2 amd64 [installed,automatic]
libpolkit-agent-1-0/stable,now 122-3 amd64 [installed,automatic]

Now, while typing this email all keyring PIDs have vanished!

$ ps ax -u nate | grep "agent\|keyring"
2151 ? Ssl 0:00 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr
2157 ? Ss 0:00 ssh-agent -D -a /run/user/1000/openssh_agent
22418 pts/0 S+ 0:00 grep --color=auto agent\|keyring

I am flummoxed.

TIA

- Nate

--
"The optimist proclaims that we live in the best of all
possible worlds. The pessimist fears this is true."
Web: https://www.n0nb.us
Projects: https://github.com/N0NB
GPG fingerprint: 82D6 4F6B 0E67 CD41 F689 BBA6 FB2C 5130 D55A 8819

[Max Nikulin]

On 14/08/2023 07:30, Nate Bargmann wrote:
> I have been using the GNOME keyring applet to manage the SSH public key
> passwords I use as it prompts to save passwords and then lets me SSH to
> other hosts without out a password prompt.

I do not know how it is arranged in Gnome, but I hope my observations
still might be helpful.

> systems on my LAN and was greeted with a password prompt for the
> corresponding public key

To be precise, it is the passphrase do decrypt your *private* key. A
public key may be known to anybody. Private key is a secret that allows
to prove that you have it without disclosing of this private key.
Encryption using a passphrase is a means to mitigate consequences if the
private key is stolen.

An ssh agent opens a socket and exposes its location through an
environment variable (perhaps using "systemctl set-environment", but I
am not sure). Try

env | grep SSH

There are multiple implementations of SSH agents: openssh, gpg,
keepassxc(?), perhaps gnome has more (seahorse? I am unsure concerning
the role of secrets service). It may happen that in your case multiple
instances are running:

/usr/lib/systemd/user/ssh-agent.service
/usr/lib/systemd/user/gpg-agent-ssh.socket
/etc/X11/Xsession.d/90x11-common_ssh-agent

GUI prompt may be just a proxy to an actual ssh agent. It just listens
its socket and displaying a password prompt on demand and passing other
messages literally.

> Now, while typing this email all keyring PIDs have vanished!

It may be a way to minimize RAM usage. The agent may be a
socket-activated process.

systemctl --user list-sockets

Check owner of $SSH_AUTH_SOCK using ss or lsof. It may give some clue
what is really happening in your case.

I suggest you to add "f" option to "ps" to see process tree. It may help
to find details concerning starting of particular agent.

ps xwwf

P.S. At certain moment gnome designers decided that password prompt must
be a modal dialog completely blocking interaction with any other
applications (including 3rd party password manager). For me it was
another reason to avoid gnome. I am aware that X11 protocol allows to
sniff keyboard events and a measure against it is grabbing input.
However I believe mouse still may be a way to call an external password
manager. After all, there are may be an option to temporary suspend
keyboard grabbing. I learned about multiple ways to start a ssh agent
during initialization of user session when I was trying to figure out
which way GUI prompt is implemented and if a more flexible dialog could
be used instead.

[Nate Bargmann]

* On 2023 14 Aug 21:29 -0500, Max Nikulin wrote:
> On 14/08/2023 07:30, Nate Bargmann wrote:
> > Now, while typing this email all keyring PIDs have vanished!
>
> It may be a way to minimize RAM usage.

I don't think so. It has been persistent in the past in Buster and
Bullseye with GNOME and is persistent on the laptop which is also
running Bookworm and GNOME. On this desktop it will rather reliably
shutdown/crash about exactly an hour after logging in with no other
desktop activity, i.e. not opening browsers or other apps.

> The agent may be a socket-activated
> process.
>
> systemctl --user list-sockets

The lists are virtually identical between the laptop:

$ systemctl --user list-sockets
LISTEN UNIT ACTIVATES
/run/user/1000/bus dbus.socket dbus.service
/run/user/1000/gcr/ssh gcr-ssh-agent.socket gcr-ssh-agent.service
/run/user/1000/gnupg/S.dirmngr dirmngr.socket dirmngr.service
/run/user/1000/gnupg/S.gpg-agent gpg-agent.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.browser gpg-agent-browser.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.extra gpg-agent-extra.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.ssh gpg-agent-ssh.socket gpg-agent.service
/run/user/1000/keyring/control gnome-keyring-daemon.socket gnome-keyring-daemon.service
/run/user/1000/pipewire-0 pipewire.socket pipewire.service
/run/user/1000/pk-debconf-socket pk-debconf-helper.socket pk-debconf-helper.service
/run/user/1000/pulse/native pipewire-pulse.socket pipewire-pulse.service

11 sockets listed.
Pass --all to see loaded but inactive sockets, too.

and the desktop:

$ systemctl --user list-sockets
LISTEN UNIT ACTIVATES
/run/user/1000/bus dbus.socket dbus.service
/run/user/1000/gcr/ssh gcr-ssh-agent.socket gcr-ssh-agent.service
/run/user/1000/gnupg/S.dirmngr dirmngr.socket dirmngr.service
/run/user/1000/gnupg/S.gpg-agent gpg-agent.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.browser gpg-agent-browser.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.extra gpg-agent-extra.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.ssh gpg-agent-ssh.socket gpg-agent.service
/run/user/1000/keyring/control gnome-keyring-daemon.socket gnome-keyring-daemon.service
/run/user/1000/pipewire-0 pipewire.socket pipewire.service
/run/user/1000/pk-debconf-socket pk-debconf-helper.socket pk-debconf-helper.service
/run/user/1000/pulse/native pipewire-pulse.socket pipewire-pulse.service
/run/user/1000/snapd-session-agent.socket snapd.session-agent.socket snapd.session-agent.service

12 sockets listed.
Pass --all to see loaded but inactive sockets, too.

On the desktop gnome-keyring-daemon has not been running for several hours.

> Check owner of $SSH_AUTH_SOCK using ss or lsof. It may give some clue what
> is really happening in your case.

On both systems that environment variable is:

$ echo $SSH_AUTH_SOCK
/run/user/1000/keyring/ssh

> I suggest you to add "f" option to "ps" to see process tree. It may help to
> find details concerning starting of particular agent.

At this point I know the agent will be working normally when I first log
into gnome-shell. This has been a reliable way to get it started. I
posted to the GNOME discourse about this and was advised to open
separate issues in the keyring Gitlab repository. They are:

https://gitlab.gnome.org/GNOME/gnome-keyring/-/issues/135
"gnome-keyring-daemon shutting down on Debian 12 shortly after logging
into GNOME Shell "

https://gitlab.gnome.org/GNOME/gnome-keyring/-/issues/136
"gnome-keyring-daemon fails to restart properly on Debian 12 "

Last night I did some testing with gdb and put my results in issue #135.
In this case the daemon crashed when I logged out of another system,
well short of the hour it will run if left idle.

[Nate Bargmann]

Well, it appears like most things in life this one was self inflicted.
🤬

Yesterday I was working on another project and to verify something was
occurring the 'strace' utility was recommended. It dawned on me that
this could help me get a clue as to what was happening to the
gnome-keyring-daemon. Using strace attached to the PID of the daemon
after a reboot showed it was getting the SIGTERM signal at exactly the
top of the hour. What?!!

After seeing this twice this morning I recalled that I have a cron entry
to kill the 'rec' program. This was to break up audio files into hourly
segments when recording an amateur radio event. This was the cron
command:

# Rotate sound recorder files
00 * * * * /usr/bin/pkill -f rec > /dev/null 2> /dev/null

On a hunch I commented that line and Voila! the daemon ran through the
next hour change and is still running as expected. The man page states
that the '-f' option matches against the full command line, not just the
process name. So, looking at the gnome-keyring-daemon command line:

1857 ? SLsl 0:00 /usr/bin/gnome-keyring-daemon --foreground --components=pkcs11,secrets --control-directory=/run/user/1000/keyring

I see that the 'rec' in 'directory' provided the match! Confirmed with
pgrep:

$ pgrep -f rec
1857

It looks like the solution for the future will be to change the cron
line to:

00 * * * * /usr/bin/pkill -f /usr/bin/rec > /dev/null 2> /dev/null

When I want to use it next in order to protect other processes.

I certainly hope this is resolved. OTOH, it forced me to recall a
number of passwords! 🤣

[David Wright]

On Mon 19 Feb 2024 at 13:26:17 (-0600), Nate Bargmann wrote:
>
> After seeing this twice this morning I recalled that I have a cron entry
> to kill the 'rec' program. This was to break up audio files into hourly
> segments when recording an amateur radio event. This was the cron
> command:
>
> # Rotate sound recorder files
> 00 * * * * /usr/bin/pkill -f rec > /dev/null 2> /dev/null
>
> On a hunch I commented that line and Voila! the daemon ran through the
> next hour change and is still running as expected. The man page states
> that the '-f' option matches against the full command line, not just the
> process name. So, looking at the gnome-keyring-daemon command line:
>
> 1857 ? SLsl 0:00 /usr/bin/gnome-keyring-daemon --foreground --components=pkcs11,secrets --control-directory=/run/user/1000/keyring
>
> I see that the 'rec' in 'directory' provided the match! Confirmed with
> pgrep:
>
> $ pgrep -f rec
> 1857
>
> It looks like the solution for the future will be to change the cron
> line to:
>
> 00 * * * * /usr/bin/pkill -f /usr/bin/rec > /dev/null 2> /dev/null

I can't get that to work here. When I kill rec, it just dies. Is pkill
sending SIGTERM, which appears to be the default? Nor can I find this
documented—though the sox docs are lengthy, so I might have missed it.

I can use SIGUSR1 with arecord, and that works perfectly.

Cheers,
David.

[Nate Bargmann]

It gets restarted by a script called by the 'tlf' amateur radio logging
program. The script is:

https://github.com/Tlf/tlf/blob/master/scripts/soundlog

It's a hack!