Bug#1030394: Cannot look-up eln file as no source file was found

[Dan Jacobson]

Package: elpa-csv-mode
Version: 1.21-1

When I open a .csv file, I see:

Warning (comp): Cannot look-up eln file as no source file was found for /usr/share/emacs/site-lisp/elpa/csv-mode-1.20/csv-mode.elc Disable showing Disable logging

That's odd.
*needs .el, so perhaps ship it.
*1.20 vs. 1.21 problem too.

[David Bremner]

Dan Jacobson <jid...@jidanni.org> writes:

> Package: elpa-csv-mode
> Version: 1.21-1
>
> When I open a .csv file, I see:
>
> Warning (comp): Cannot look-up eln file as no source file was found
> for /usr/share/emacs/site-lisp/elpa/csv-mode-1.20/csv-mode.elc Disable
> showing Disable logging

That file is not present on my system when installing version
1.21-1. Nor are any elc files left behind when removing
elpa-csv-mode. So I don't know how to reproduce this; it seems a
previous package upgrade / removal must have partially failed (or been
interrupted somehow). Let me know if you have a recipe to reproduce this
on a clean system (e.g. in a chroot).

If the file /usr/share/emacs/site-lisp/elpa/csv-mode-1.20/csv-mode.elc
is present after you remove the package elpa-csv-mode, it should be safe
to delete it by hand.

d

[Dan Jacobson]

reopen 1030394
retitle 1030394 1.20 not cleaned up
thanks

Wait, on my other computer there is the exact same problem:

$ ls -log /usr/share/emacs/site-lisp/elpa/
drwxr-xr-x 2 4096 2018-08-07 circe-2.6
drwxr-xr-x 2 4096 2022-05-26 csv-mode-1.20
drwxr-xr-x 2 4096 01-25 12:31 csv-mode-1.21
drwxr-xr-x 2 4096 01-25 12:31 debian-el-37

[Nicholas D Steeves]

reassign 1030394 dh-elpa
retitle 1030394 dh-elpa: elpa-csv-mode 1.20 not cleaned up
tags 1030394 - moreinfo - unreproducible
thanks

These bugs are the same type as #1040787. Thus far, the only trigger
that we've been able to identify is a dist-upgrade (or full-upgrade)
from bullseye to bookworm. I ran these in a 'su -' environment on real
machines. It may be that the actual trigger condition is
buster2bullseye2bookworm. #1040690, which is currently filed against
emacs-el (with an "affects dh-elpa") appears to have the most activity
as well as user-confirmed reproducibility.

Note that this is a disruptive bug to users, who are having to resort to
heavy-handed methods.

To all affected users: Do you remember if you ever manually installed an
affected elpa-package from sid/unstable or from testing? I'm curious if
this might be part of the trigger condition. Likewise, do you remember
if you installed dh-elpa from backports? While I think both of these
cases are unlikely to have caused problems, one might as well be
thorough!

Regards,
Nicholas

[Richard Lewis]

On Fri, 21 Jul 2023 at 23:39, Nicholas D Steeves <st...@debian.org> wrote:

> retitle 1030394 dh-elpa: elpa-csv-mode 1.20 not cleaned up

important to note that it's not just this one package, but many elpa
packages (but not all) which were either upgraded or purged as part of
the upgrade: I get the same set of warnings on two separate systems.
(I have elpa-csv-mode and no issue, but i don't know if i had it
installed in bullseye)

Warnings i have:

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/helpful-0.18/helpful.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/elisp-refs-1.3/elisp-refs.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/dash-2.17.0/dash.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/loop-1.3/loop.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/dash-functional-1.2.0/

But i also have duplicate subdirectories in
/usr/share/emacs/site-lisp/elpa for dash,elisp-refs,git-commit,
helpful, magit and with-editor
Lots of broken symlinks, eg
/usr/share/emacs/site-lisp/elpa/transient-0.2.0.30 has
transient-autoloads.el ->
/usr/share/emacs/site-lisp/elpa-src/transient-0.2.0.30//transient-autoloads.el
which doesnt exist
Lots of 'empty' (32-byte) Install.log.gz files such as
/usr/share/emacs/site-lisp/elpa/rainbow-delimiters-2.1.5/Install.log.gz
-- which unzul to an empty file

Current versions of packages
elpa-helpful:all/bookworm 0.19-2 uptodate
elpa-elisp-refs:all/bookworm 1.4-1 uptodate
elpa-dash:all/bookworm 2.19.1+git20220608.1.0ac1ecf+dfsg-1 uptodate
elpa-loop:all not installed # was installed as a recommends of
something in bullseye i assume
elpa-csv-mode:all/bookworm 1.22-1 uptodate # i dont know if i had this
via dependencies in bullseye

> To all affected users: Do you remember if you ever manually installed an
> affected elpa-package from sid/unstable or from testing? I'm curious if
> this might be part of the trigger condition. Likewise, do you remember
> if you installed dh-elpa from backports? While I think both of these
> cases are unlikely to have caused problems, one might as well be
> thorough!

I have never installed anything emacs-related from anywhere other than stable.
All systems have only run stable, and have been upgraded from stable
to stable (from stretch i think).

While I havn't tried to make it reproducible, i did reproduce it in
the sense that two separate systems which had the same set of
emacs-related packages installed ended up with the same warnings. I
have the upgrade log for one - are there bits that are helpful to look
at - i didnt see anything complain, but the new emacs and new dh-elpa
were unpacked before the other elpa- packages (i think)

I can see dh-elpa tried to purge the old files, but presumably this
did not actualy work, although there is no error message:

dh-elpa: purging flavor specific files for emacs^M
Remove elpa-helpful for emacs^M
remove/helpful-0.19: Handling removal of emacsen flavor emacs^M
dh-elpa: purging flavor specific files for emacs^M

(i was surprised it says removing 0.19 not 0.18 but all the other elpa
packages have the new version here too.
this bit came after new emacs and dh-elpa were unpacked but before
they were 'Set up' (configured?))

Later there is some tsort issue,

Setting up elpa-helpful (0.19-2) ...^M
tsort: -: input contains a loop:^M
tsort: elpa-dash^M
tsort: emacsen-common^M
tsort: -: input contains a loop:^M
tsort: emacsen-common^M
tsort: elpa-s^M


My emacs-related packages: all would have been at the previous
bullseye point-release/security upgrade before upgrading:
emacs-gtk, emacs-el
emacs-common-non-dfsg, emacs-goodies-el, bbdb3, elpa-debian-el,
elpa-dpkg-dev-el, elpa-org, elpa-org-bullets, elpa-company,
elpa-dumb-jump, elpa-flycheck, elpa-helpful, elpa-systemd,
elpa-magit, elpa-ag, elpa-rainbow-delimiters, elpa-rich-minority

(Plus all recommends. And a couple of locally packaged variants -
which were not changed on upgrade but did use dh-elpa).
I used aptitude to upgrade to bookworm, and set anything removed to be purged.

[Richard Lewis]

An attempt to reproduce - partially successful, maybe reveals deeper issues!

su -
mkdir /tmp/bullseye
cd /tmp/bullseye
debootstrap bullseye . https://deb.debian.org/debian

chroot . apt install emacs elpa-helpful

sed -i s/bullseye/bookworm/ ./etc/apt/sources.list
chroot . apt update
chroot . apt upgrade

# This fails with:
Setting up tasksel-data (3.73) ...
[15/1797]
Setting up emacs-common (1:28.2+1-15) ...
Setting up emacs-bin-common (1:28.2+1-15) ...
Setting up emacs-gtk (1:28.2+1-15) ...

tsort: -: input contains a loop:

tsort: elpa-dash
tsort: emacsen-common

tsort: -: input contains a loop:

tsort: elpa-s
tsort: emacsen-common
Install elpa-dash for emacs
install/dash-2.19.1: Handling install of emacsen flavor emacs
install/dash-2.19.1: byte-compiling for emacs
>>Error occurred processing dash.el: File error (("Doing chmod" "Operation not supported" "/usr/share/emacs/site-lisp/elpa/dash-2.19.1/dash.
elceCdFwM"))
ERROR: install script from elpa-dash package failed
dpkg: error processing package emacs-gtk (--configure):
installed emacs-gtk package post-installation script subprocess
returned error exit status 1
dpkg: dependency problems prevent configuration of emacs:
emacs depends on emacs-gtk (>= 1:27.1) | emacs-lucid (>= 1:27.1) |
emacs-nox (>= 1:27.1); however:
Package emacs-gtk is not configured yet.
Package emacs-lucid is not installed.
Package emacs-nox is not installed.
dpkg: error processing package emacs (--configure):
dependency problems - leaving unconfigured
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for libc-bin (2.36-9+deb12u1) ...
Processing triggers for debianutils (5.7-0.4) ...
Processing triggers for install-info (6.8-6+b1) ...
Processing triggers for ca-certificates (20230311) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Processing triggers for libgdk-pixbuf-2.0-0:amd64 (2.42.10+dfsg-1+b1) ...
Errors were encountered while processing:
emacs-gtk
emacs
E: Sub-process /usr/bin/dpkg returned an error code (1)

BUT, /tmp/bullseye/usr/share/emacs/site-lisp/elpa/helpful-0.18/ has
the broken symlinks - and presumably should not exist at all. so maybe
this helps narrow it down!

[David Bremner]

Richard Lewis <richard.le...@googlemail.com> writes:

> An attempt to reproduce - partially successful, maybe reveals deeper issues!
>
> su -
> mkdir /tmp/bullseye
> cd /tmp/bullseye
> debootstrap bullseye . https://deb.debian.org/debian
>
> chroot . apt install emacs elpa-helpful
>
> sed -i s/bullseye/bookworm/ ./etc/apt/sources.list
> chroot . apt update
> chroot . apt upgrade

Thanks Richard that does seem like progress.

I chrooted into to "bullseye" after the upgrade failure, and narrowed
the failing command down to

$ emacs --quick --batch -l package --eval "(setq package-user-dir \"/nonexistent\")" --eval "(add-to-list 'package-directory-list \"/usr/share/emacs/site-lisp/elpa-src\")" -f package-initialize -f batch-byte-compile *.el

I'm not sure why that command should be be failing, certainly I can
chmod the file manually (as root).

[David Bremner]

David Bremner <da...@tethera.net> writes:

>>
>> chroot . apt install emacs elpa-helpful

Try the same set of steps without elpa-helpful, and the upgrade still
fails with

Install emacsen-common for emacs
emacsen-common: Handling install of emacsen flavor emacs
>>Error occurred processing /usr/share/emacs/site-lisp/debian-s
tartup.el: File error (("Doing chmod" "Operation not supported"
"/usr/share/emacs/site-lisp/debian-startup.elc90iTVh"))

At this point dh-elpa is not involved. It seems most likely to be an
Emacs 28 bug, but I guess it could also be an issue with
emacsen-common.

[Richard Lewis]

I suspect a plain chroot isnt 'enough', i had success with systemd-nspawn:

ln -s /tmp/bullseye/ /var/lib/machines

# im sure there is a better way than these two lines
cp /etc/passwd bullseye/etc/passwd
cp /etc/shadow bullseye/etc/shadow

systemd-nspawn --ephemeral --boot --machine bullseye
# (you dont really need --ephemeral of course)

log into the container as root:
# apt install emacs-gtk

(and say yes to allow the installation to finish)

# ls /usr/share/emacs/site-lisp/elpa
dash-2.17.0 dash-functional-1.2.0 elisp-refs-1.4 helpful-0.18 loop-1.3
dash-2.19.1 elisp-refs-1.3 f-0.20.0 helpful-0.19 s-1.12.0

Shows the 'duplicate' dirs for the old versions of elpa-dash and elpa-helpful

Additionally:
# echo "(require 'helpful nil t)" > ~/.emacs
# emacs --no-init-file --batch --eval "(byte-compile-file \".emacs\")"

# reproduces the warnings:

Loading /etc/emacs/site-start.d/00debian.el (source)...

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/helpful-0.18/helpful.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/elisp-refs-1.3/elisp-refs.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/dash-2.17.0/dash.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/loop-1.3/loop.elc

Warning (comp): Cannot look-up eln file as no source file was found

for /usr/share/emacs/site-lisp/elpa/dash-functional-1.2.0/dash-functional.elc

## to cleanup:
# close the container by pressing C-] several times
# rm /var/lib/machines/bullseye and /tmp/bullseye

[Christoph Groth]

Nicholas D Steeves wrote:

> To all affected users: Do you remember if you ever manually installed
> an affected elpa-package from sid/unstable or from testing? I'm
> curious if this might be part of the trigger condition.

Before the upgrade to bookworm I was running an almost pure bullseye.
The exception were a few elpa packages from bookworm (i.e. then
testing). We can try to reconstruct the list of these package from the
output of the following command:

zgrep -h ' elpa-' /var/log/dpkg.log* | grep ' upgrade '

2023-07-20 20:31:46 upgrade elpa-adaptive-wrap:all 0.8-1 0.8-3
2023-07-20 20:31:46 upgrade elpa-websocket:all 1.13-1 1.13-3
2023-07-20 20:31:47 upgrade elpa-atomic-chrome:all 2.0.0-2 2.0.0-4
2023-07-20 20:31:47 upgrade elpa-dash:all 2.19.1+dfsg-1 2.19.1+git20220608.1.0ac1ecf+dfsg-1
2023-07-20 20:31:47 upgrade elpa-emacsql:all 3.0.0+ds-2 3.1.1+ds-1
2023-07-20 20:31:48 upgrade elpa-git-commit:all 3.3.0-1 3.3.0-2
2023-07-20 20:31:48 upgrade elpa-htmlize:all 1.55-1 1.56-1
2023-07-20 20:31:49 upgrade elpa-magit-section:all 3.3.0-1 3.3.0-2
2023-07-20 20:31:49 upgrade elpa-magit:all 3.3.0-1 3.3.0-2
2023-07-20 20:31:50 upgrade elpa-markdown-mode:all 2.4-1 2.5-1
2023-07-20 20:31:51 upgrade elpa-notmuch:all 0.31.4-2 0.37-1
2023-07-20 20:31:52 upgrade elpa-org:all 9.4.0+dfsg-1 9.5.2+dfsh-5
2023-07-20 20:31:52 upgrade elpa-transient:all 0.3.6-2 0.3.7-1
2023-07-20 20:31:53 upgrade elpa-yaml-mode:all 0.0.15-1 0.0.15-2
2023-07-20 21:19:26 upgrade elpa-emacsql-sqlite:amd64 3.0.0+ds-2 3.1.1+ds-1

(The main upgrade from bullseye to bookworm occurred around 20:30.)

Comparing the left versions in the log to the ones in bullseye one can
see that at least the following packages were from testing:

elpa-dash
elpa-git-commit
elpa-magit-section
elpa-magit
elpa-transient

To this list one should add elpa-org-roam which did not require an
upgrade during the bullseye -> bookworm transition and perhaps other
elpa packages that I lost track of.

> Likewise, do you remember if you installed dh-elpa from backports?
> While I think both of these cases are unlikely to have caused
> problems, one might as well be thorough!

I do not remember ever installing dh-elpa from any source, nor can
I find any trace of it in the logs.

[David Bremner]

Richard Lewis <richard.le...@googlemail.com> writes:

> I suspect a plain chroot isnt 'enough', i had success with systemd-nspawn:
>

Not sure what you mean here. The reproducer using chroot you posted
works fine for me, it's just that AFAICT the real bug is the emacs
upgrade failing (and more precisely, in emacs being unable to run a
simple batch-byte-compile command after the new version is installed),
nothing to do with addon packages.

[Richard Lewis]

i may be wrong, but i suspect there are two separate issues:

- a previously undetected bug where upgrading emacs in a plain chroot doesnt work. clearly emacs can be upgraded on a normal system or in a container or it would have been spotted well before now. my guess is that this will be found to be caused by emacs relying in some complex way on some feature of a "real" system not suported by plain old chroots..( i have no idea what but maybe it needs /dev/pts mounted or i think inotify might not work in a chroot). 

- upgrading emacs and dh-elpa and (some) packages made using dh-elpa doesnt properly remove the "old" directories in /usr/share/emacs. (#1040690)

it is not obvious to me that fixing the first will help with the second, because if we do everything in a systemd-nspawn container we dont hit the first but at all, but do still see the second. (and the second is the one that im really hoping someone can solve !)

of couse this is just my speculation, and may all be nonsense. 

[David Bremner]

Richard Lewis <richard.le...@googlemail.com> writes:

> I suspect a plain chroot isnt 'enough', i had success with systemd-nspawn:
>
> ln -s /tmp/bullseye/ /var/lib/machines
>
> # im sure there is a better way than these two lines
> cp /etc/passwd bullseye/etc/passwd
> cp /etc/shadow bullseye/etc/shadow
>
> systemd-nspawn --ephemeral --boot --machine bullseye
> # (you dont really need --ephemeral of course)
>
> log into the container as root:
> # apt install emacs-gtk
>
> (and say yes to allow the installation to finish)
>
> # ls /usr/share/emacs/site-lisp/elpa
> dash-2.17.0 dash-functional-1.2.0 elisp-refs-1.4 helpful-0.18 loop-1.3
> dash-2.19.1 elisp-refs-1.3 f-0.20.0 helpful-0.19 s-1.12.0
>
> Shows the 'duplicate' dirs for the old versions of elpa-dash and elpa-helpful
>

Did you start from a clean debootstrap here? Because I don't see where
in your second reproducer the addon packages get installed.

[Richard Lewis]

no, i reused the chroot from the "first attempt" reproducer

A clean recipe, not requiring any 'login' is:

mkdir -p /tmp/bullseye

cd /tmp/bullseye

debootstrap bullseye . https://deb.debian.org/debian

ln -s /tmp/bullseye /var/lib/machines/bullseye

systemd-nspawn --machine bullseye apt install emacs elpa-helpful

# at this stage we have:

# $ ls /tmp/bullseye/usr/share/emacs/site-lisp/elpa/
# dash-2.17.0  dash-functional-1.2.0  elisp-refs-1.3  f-0.20.0  helpful-0.18  loop-1.3  s-1.12.0

sed -i /bullseye/bookworm/ tmp/bullseye/apt/sources.list

#

systemd-nspawn --machine bullseye apt update

systemd-nspawn --machine bullseye apt upgrade

# this all works, including upgrading emacs :)

# but, old directories for elpa-dash and elpa-helpful and elpa-elisp-refs have not been removed:

# $ ls /tmp/bullseye/usr/share/em$ ls /tmp/bullseye/usr/share/emacs/site-lisp/elpa{,-src}

/tmp/bullseye/usr/share/emacs/site-lisp/elpa:

dash-2.17.0  dash-functional-1.2.0  elisp-refs-1.4  helpful-0.18  loop-1.3
dash-2.19.1  elisp-refs-1.3         f-0.20.0        helpful-0.19  s-1.12.0

/tmp/bullseye/usr/share/emacs/site-lisp/elpa-src:

dash-2.19.1  dash-functional-1.2.0  elisp-refs-1.4  f-0.20.0  helpful-0.19  loop-1.3  s-1.12.0

# nb: on the host /tmp/bullseye and the symlink /var/lib/machine/bullseye should be removed once finished!

[David Bremner]

Richard Lewis <richard.le...@googlemail.com> writes:

> On Sun, 23 Jul 2023, 12:34 David Bremner, <da...@tethera.net> wrote:
>
>> Did you start from a clean debootstrap here? Because I don't see where
>> in your second reproducer the addon packages get installed.
>>
>
> no, i reused the chroot from the "first attempt" reproducer
> A clean recipe, not requiring any 'login' is:
>
> mkdir -p /tmp/bullseye
> cd /tmp/bullseye
> debootstrap bullseye . https://deb.debian.org/debian
> ln -s /tmp/bullseye /var/lib/machines/bullseye

btw, there is no need to use a symlink here (depends on space in /var I guess)

> systemd-nspawn --machine bullseye apt install emacs elpa-helpful

A smaller set of packages is just emacs and elpa-dash.


> sed -i /bullseye/bookworm/ tmp/bullseye/apt/sources.list

There are a couple of typos here, but I get what you meant. Should be
more like

sed -i s/bullseye/bookworm/ /var/lib/machines/bullseye/etc/apt/sources.list

> systemd-nspawn --machine bullseye apt update
> systemd-nspawn --machine bullseye apt upgrade

I checked that changing this to full-upgrade does not change anything

> # this all works, including upgrading emacs :)

OK, this downgrades the importance of the crash when upgrading emacs in
a chroot, I agree.

As far as the actual bug with failing to clean up, I ran

% systemd-nspawn --machine bullseye /usr/lib/dh-elpa/helper/remove emacs dash 2.17.0

and that cleans up the directory

/usr/share/emacs/site-lisp/elpa/dash-2.17.0

so the bug is at some other level of the stack. I guess I will have to
look at the log from the upgrade, but I haven't had a chance to do that yet.

[Richard Lewis]

I was trying to understand when (and how ) your command above was intended to be run as part of the upgrade. I cna see that in bullseye /usr/lib/emacsen-common/packages/remove/elpa-dash would do it if called with 'emacs'. but this is never called:

What happens in the 'apt upgrade' is:

the old emacsen-common prerm is called ('<old prerm> upgrade <new-version>'):

/var/lib/dpkg/info/emacsen-common.prerm upgrade 3.0.5

This calls:

/usr/lib/emacsen-common/emacs-package-remove --prerm emacsen-common

This calls:

/usr/lib/emacsen-common/packages/remove/emacsen-common

and at the end it _unlinks_:

/var/lib/emacsen-common/state/package/installed/emacsen-common

Then, apt prepares to unpack elpa-dash:

The elpa-dash prerm (from bullseye) is called as:

/var/lib/dpkg/info/elpa-dash.prerm upgrade 2.19.1+git20220608.1.0ac1ecf+dfsg-1)

but this starts with:

if [ -e /var/lib/emacsen-common/state/package/installed/emacsen-common -a -x                                          

  /usr/lib/emacsen-common/emacs-package-remove ] ; then

/usr/lib/emacsen-common/emacs-package-remove --prerm elpa-dash                                                      
fi                            

... and so does nothing, because /var/lib/emacsen-common/state/package/installed/emacsen-common is gone.

I couldnt add debug statements to the new maintainer scripts, but: reading https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#details-of-unpack-phase-of-installation-or-upgrade

I think we are now at step 3: '<new-preinst> upgrade <old-version> <new-version>'

the elpa-dash preinst starts with

if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove"  
  ] ; then

so this does nothing.

then elpa-dash files are unpacked (step 4)

now /usr/lib/emacsen-common/packages/remove/elpa-dash contains 

'usr/lib/dh-elpa/helper/remove $1 dash 2.19.1' so we can no longer remove the old version

So the issue is that emacsen-common is unpacked before the elpa-dash package

[Nicholas D Steeves]

Richard Lewis wrote:
> David Bremner wrote:
>
> > Richard Lewis writes:

> > > David Bremner wrote:
> > >
> >
> > As far as the actual bug with failing to clean up, I ran
> >
> > % systemd-nspawn --machine bullseye /usr/lib/dh-elpa/helper/remove emacs
> > dash 2.17.0
> >
> > and that cleans up the directory
> >
> > /usr/share/emacs/site-lisp/elpa/dash-2.17.0
> >
> > so the bug is at some other level of the stack. I guess I will have to
> > look at the log from the upgrade, but I haven't had a chance to do that
> > yet.
> >
>
> I was trying to understand when (and how ) your command above was intended
> to be run as part of the upgrade. I cna see that in bullseye
> /usr/lib/emacsen-common/packages/remove/elpa-dash would do it if called
> with 'emacs'. but this is never called:
>
> What happens in the 'apt upgrade' is:
>
> the old emacsen-common prerm is called ('<old prerm> upgrade
> <new-version>'):
>
> /var/lib/dpkg/info/emacsen-common.prerm upgrade 3.0.5
>
> This calls:
> /usr/lib/emacsen-common/emacs-package-remove --prerm emacsen-common
>
> This calls:
> /usr/lib/emacsen-common/packages/remove/emacsen-common
>
> and at the end it _unlinks_:
> /var/lib/emacsen-common/state/package/installed/emacsen-common

@1

> Then, apt prepares to unpack elpa-dash:
>
> The elpa-dash prerm (from bullseye) is called as:
> /var/lib/dpkg/info/elpa-dash.prerm upgrade
> 2.19.1+git20220608.1.0ac1ecf+dfsg-1)
>
> but this starts with:
>
> if [ -e /var/lib/emacsen-common/state/package/installed/emacsen-common -a
> -x
> /usr/lib/emacsen-common/emacs-package-remove ] ; then
> /usr/lib/emacsen-common/emacs-package-remove --prerm elpa-dash
>
> fi
>
> ... and so does nothing,
> because /var/lib/emacsen-common/state/package/installed/emacsen-common is
> gone.

Oh! It's a state-related bug! I wonder if emacsen-common (old version)
has been deinstalled at @1 (see above), or if the state is
emacsen-common (new version) is unpacked, but unconfigured, and thus
missing /var/lib/emacsen-common/state/package/installed/emacsen-common,
which is presumably created as a last step during package configuration
post-unpack?

In other words, elpa-dash (and other...most?..all? dh-elpa-using
packages) upgrades depends on having emacsen-common fully installed and
configured at @1.

I wonder if this can be solved in emacsen-common, or if it needs to be
solved in dh-elpa and then all the dh-elpa-using packages rebuilt to
generate new prerm scripts? Which do you think would be the better
approach?

Cheers,
Nicholas

[Richard Lewis]

On Thu, 17 Aug 2023, 20:57 Nicholas D Steeves, <st...@debian.org> wrote:

Richard Lewis wrote:
> David Bremner wrote:
> > Richard Lewis writes:
> > > David Bremner wrote:

> What happens in the 'apt upgrade' is:
>
> the old emacsen-common prerm is called ('<old prerm> upgrade
> <new-version>'):

> and at the end it _unlinks_:
> /var/lib/emacsen-common/state/package/installed/emacsen-common

@1

> Then, apt prepares to unpack elpa-dash:
>
> The elpa-dash prerm (from bullseye) is called as:
> /var/lib/dpkg/info/elpa-dash.prerm upgrade
> 2.19.1+git20220608.1.0ac1ecf+dfsg-1)
>
> but this starts with:
>
> if [ -e /var/lib/emacsen-common/state/package/installed/emacsen-common -a
> -x
>   /usr/lib/emacsen-common/emacs-package-remove ] ; then
> /usr/lib/emacsen-common/emacs-package-remove --prerm elpa-dash
>
> fi
>
> ... and so does nothing,
> because /var/lib/emacsen-common/state/package/installed/emacsen-common is
> gone.

Oh!  It's a state-related bug!  I wonder if emacsen-common (old version)
has been deinstalled at @1 (see above), or if the state is
emacsen-common (new version) is unpacked, but unconfigured, and thus
missing /var/lib/emacsen-common/state/package/installed/emacsen-common,
which is presumably created as a last step during package configuration
post-unpack?

the latter - new emacsen-common is present but in state 'unpacked' before new dh-* is unpackaged

In other words, elpa-dash (and other...most?..all? dh-elpa-using
packages) upgrades depends on having emacsen-common fully installed and
configured at @1.

this is  my understanding yes:  this is a feature/assumption made by the current design but not enforced through package dependencies or code in apt

I wonder if this can be solved in emacsen-common, or if it needs to be
solved in dh-elpa and then all the dh-elpa-using packages rebuilt to
generate new prerm scripts?  Which do you think would be the better
approach?

I would guess the former is possible. but I think what is missing is clear documentation of what features all the machinery with emacsen-common and 'flavors' and dh-elpa is trying to provide

( i suspect dpkg triggers should be the answer- should the question be understood better --- eg it looks like jed does something similar with .sl files, at least i think it does -- i didnt find any documentation on that either)

[Rob Browning]

Richard Lewis <richard.le...@googlemail.com> writes:

> the latter - new emacsen-common is present but in state 'unpacked' before
> new dh-* is unpackaged

I think I may understand what's going on now, and if so, the issue
rests on a misunderstanding in emacsen-common about how the
maintainerscript states work.

We've been discussing some improvements, some more dramatic than others
on #debian-emacs, and I've been toying with some of the possibilities.

> ( i suspect dpkg triggers should be the answer- should the question be
> understood better --- eg it looks like jed does something similar with .sl
> files, at least i think it does -- i didnt find any documentation on that
> either)

Hah, that's one of the approaches I've been toying with for the past few
days. If it pans out, I'll likely make a proposal along with changes to
debian-emacs-policy in a while.

I'd considered triggers a few times in the past, but only recently
thought of a way that addresses the issues I'd previously been concerned
about.

If this does work out, we're also likely to add recompilation of
rdepends during add-on package upgrades (i.e. in case "public" macros
have changed in incompatible ways).

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4