Re: How can I find packages manually installed using "dpkg -i"?

[Marco M.]

Am 02.10.2023 um 10:05:46 Uhr schrieb Ottavio Caruso:

> Before you say:
>
> $ apt list '?narrow(?installed, ?not(?origin(Debian)))'
>
> The problem with that is there are packages that I added from the
> Linux Mint repos (not manually) and that I want to keep and they all
> have the tag "local". For example:
>
> mintmenu/now 6.1.5 all [installed,local]
> mintreport/now 1.3.5 all [installed,local]
> mintsources/now 2.1.9 all [installed,local]

That means it cannot be found in the currently enables repos.

Do you want to list such packages or the packages that are listed as
manually installed (and not automatically because another package
depends on them)?

[to...@tuxteam.de]

On Mon, Oct 02, 2023 at 10:05:46AM +0000, Ottavio Caruso wrote:
> I want to upgrade Bullseye to Bookworm and I want to remove all packages
> that I installed manually, downloading the .debs
> and then using "dpkg -i".

[...]

If you're lucky, /var/log/dpkg.log in combination with one or both of
/var/log/apt/history.log and/or /var/log/apt/term.log and some elbow
grease (or some creative scripting) might help.

> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?

How true. However, people who need to understand this are most
likely to not read that far :-(

Cheers
--
t

[Max Nikulin]

On 02/10/2023 17:05, Ottavio Caruso wrote:
> Before you say:
>
> $ apt list '?narrow(?installed, ?not(?origin(Debian)))'
>
> The problem with that is there are packages that I added from the Linux
> Mint repos (not manually) and that I want to keep and they all have the
> tag "local". For example:
>
> mintmenu/now 6.1.5 all [installed,local]
> mintreport/now 1.3.5 all [installed,local]
> mintsources/now 2.1.9 all [installed,local]

Apt does not store in its DB from which sources particular packages were
installed. Filters are based on presence of packages in currently
configured repositories. See apt-patterns(7) for various available
options such as ~M for automatically installed packages. I would try

apt list '~o!~M'

To get list of local manually installed packages. If the number of them
is not high you may remove them and then run

apt autoremove --purge

Notice that packages from non-standard repositories may break upgrade
due to their dependencies.

[The Wanderer]

On 2023-10-02 at 09:28, Ottavio Caruso wrote:

> Am 02/10/2023 um 10:12 schrieb Marco M.:
>
>> That means it cannot be found in the currently enables repos.
>>
>> Do you want to list such packages
>

> Yeah, the one for which I had to manually use "dpkg -i".

That information is not tracked.

What is tracked is "the package versions known to be available from each
registered repository" and "the package versions which are installed".
Whether the package version that is installed came from one of the
registered repositories, much less whether it was installed from that
repository, is not tracked.

Unless I'm much mistaken, if you installed one package manually (i.e.,
not as a dependency) from a repository and another package locally via
'dpkg -i', the installation state of those two packages will look the
same. The system does not track anything which would let it tell you
where the package came from; at most, if the installed version of a
package is available from a currently-registered repository, it may be
able to tell you where you could get that package from again.

It's entirely probable that there is no way to identify the set of
packages you're interested in, short of a combination of manual
archaeology in local log files (and the local apt package cache) and
relying on your own memory.

--
The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw

[Marco M.]

Am 02.10.2023 um 13:28:05 Uhr schrieb Ottavio Caruso:

> Yeah, the one for which I had to manually use "dpkg -i".

I don't know a way to only show them. Every package has the attribute
"automatically installed". Every package you manually installed doesn't
have that.

[Greg Wooledge]

On Mon, Oct 02, 2023 at 09:43:39AM -0400, The Wanderer wrote:
> On 2023-10-02 at 09:28, Ottavio Caruso wrote:
> > Yeah, the one for which I had to manually use "dpkg -i".
>
> That information is not tracked.
>
> What is tracked is "the package versions known to be available from each
> registered repository" and "the package versions which are installed".

There *is* tracking. Packages can be marked as "automatically installed"
or not. The problem is, the marking is not consistent with user
expectations.

>From apt-patterns(7):

?automatic, ~M
Selects packages that were installed automatically.

[...]

EXAMPLES
[...]
apt list '~i !~M (~slibs|~sperl|~spython)'
List all manually-installed packages in sections matching libs,
perl, or python.

But if you try this locally, you'll discover that a bunch of packages
are listed which you don't expect.

unicorn:~$ apt list '~i !~M'
Listing... Done
abcde/stable,stable,now 2.9.3-1 all [installed]
acl/stable,now 2.3.1-3 amd64 [installed]
adduser/stable,stable,now 3.134 all [installed]
alsa-utils/stable,now 1.2.8-1 amd64 [installed]
an/stable,now 1.2-7+b1 amd64 [installed]
apt-listchanges/stable,stable,now 3.24 all [installed]
apt-transport-https/stable,stable,now 2.6.1 all [installed]
apt-utils/stable,now 2.6.1 amd64 [installed]
apt/stable,now 2.6.1 amd64 [installed]
aptitude/stable,now 0.8.13-5 amd64 [installed]
at/stable,now 3.2.5-1+b1 amd64 [installed]
[...]

I didn't need to install "adduser" or "apt" manually, for example, and
yet they're listed here. As it turns out, most (or all?) of the packages
that were installed by the installer *also* show up as manually installed.
This makes it unsuitable for most people's purposes.

[The Wanderer]

On 2023-10-02 at 09:52, Greg Wooledge wrote:

> On Mon, Oct 02, 2023 at 09:43:39AM -0400, The Wanderer wrote:
>
>> On 2023-10-02 at 09:28, Ottavio Caruso wrote:
>>
>>> Yeah, the one for which I had to manually use "dpkg -i".
>>
>> That information is not tracked.
>>
>> What is tracked is "the package versions known to be available from
>> each registered repository" and "the package versions which are
>> installed".
>
> There *is* tracking. Packages can be marked as "automatically
> installed" or not. The problem is, the marking is not consistent
> with user expectations.

That does exist, yes, but it tracks "explicitly selected for
installation" vs. "installed as a dependency" - and even beyond the
problems which you note, that isn't the distinction which Ottavio
appears to be interested in.

The distinction Ottavio is interested in appears to be "installed by
'dpkg -i' from a .deb file on the command line" vs. "installed via 'apt'
or 'apt-get' from a repository". (It's not clear how "installed via
'apt' from a .deb file on the command line" would be counted for this
purpose.)

As far as I'm aware, there is no tracking of where the package entered
the local system from, which is what would be necessary in order for
something to report the information Ottavio appears to be seeking.

[to...@tuxteam.de]

On Mon, Oct 02, 2023 at 09:52:39AM -0400, Greg Wooledge wrote:
> On Mon, Oct 02, 2023 at 09:43:39AM -0400, The Wanderer wrote:
> > On 2023-10-02 at 09:28, Ottavio Caruso wrote:
> > > Yeah, the one for which I had to manually use "dpkg -i".
> >
> > That information is not tracked.
> >
> > What is tracked is "the package versions known to be available from each
> > registered repository" and "the package versions which are installed".
>
> There *is* tracking. Packages can be marked as "automatically installed"
> or not. The problem is, the marking is not consistent with user
> expectations.

But not if you use dpkg directly (dpkg doesn't even resolve dependencies,
but just gives up when some aren't met). That's why I proposed comparing
apt logs and dpkg logs -- things in the latter but not in the former were
probably installed with straight dpkg.

It probably ain't straightforward, though. And the logs might have been
rotated out anyway.

Cheers
--
t

[David Wright]

If you have complete logs and try this, presumably coming up with a
sorted list of apt-installed packages (remembering --unique) from its
history, and a similar list from the ' install ' lines in dpkg.log*,
bear in mind that you need to ignore the dpkg.log until about when
locales is installed, as APT never sees a load of packages installed
before that point.

Also be careful if you're tempted to sort dpkg.log's install lines
by time, instead of grepping the logs in the correct (non-collating)
order, because for people in the western hemisphere, the packages you
want to ignore will have UTC timestamps, placing them in the midst
of packages installed after the d-i has finished. (The timestamps
jump from UTC to localtime after locales is installed.)

To avoid all these complications in future, I would suggest that the
OP (and anybody else) modify their logrotate configuration files thus:

$ grep rotate /etc/logrotate.d/{apt,dpkg}
/etc/logrotate.d/apt: rotate -1
/etc/logrotate.d/apt: rotate -1
/etc/logrotate.d/dpkg: rotate -1
$

and jettison dpkg -i in favour of: apt install /path/to/some.deb
where the /path/to/ is essential even if it's only ./some.deb.

With those changes, it will be easy to spot these particular packages
just by grepping .deb$ (strictly, \.deb$) in …/apt/history.log*.

Cheers,
David.

[to...@tuxteam.de]

On Mon, Oct 02, 2023 at 11:24:04PM -0500, David Wright wrote:

[...]

> If you have complete logs and try this, presumably coming up with a
> sorted list of apt-installed packages (remembering --unique) from its
> history, and a similar list from the ' install ' lines in dpkg.log*,
> bear in mind that you need to ignore the dpkg.log until about when
> locales is installed, as APT never sees a load of packages installed
> before that point.

[...]

Very good points you make, all of them. Thanks.

>
> Also be careful if you're tempted to sort dpkg.log's install lines
> by time, instead of grepping the logs in the correct (non-collating)
> order, because for people in the western hemisphere, the packages you
> want to ignore will have UTC timestamps, placing them in the midst
> of packages installed after the d-i has finished. (The timestamps
> jump from UTC to localtime after locales is installed.)

Eek. And as I've seen they have no time offset indicators, so
the times even can be ambiguous (winter to summer transition).
I always thought it's a bad idea to use anything else than UTC
in logs.

Cheers
--
t

[Mike Castle]

Some tools I've been using lately are apt-mark and "dpkg-query --show".

The following UNTESTED commands (ran as a normal user):

(apt-mark showauto ; apt-mark showmanual) > apt-thinks-you-installed.txt
dpkg-query --show --showformat='${Package}\n' | grep -v -F
apt-thinks-you-installed.txt > rest.txt

The file "rest.txt" should have a list of packages installed that were
NOT installed via apt. With any luck, it is small enough to examine
manually.

You could do something like "apt list" to get a list of all packages
known by apt and see if you'd prefer to use just use the Debian
instead of Mint versions. And anything not in that list *probably*
came from other manual sources and you can do what you will with that
information.

You could poke around in /var/lib/apt/lists/ and see if the files from
the mint repos you used in the past are still there (I don't know if
they get cleaned up or not, might get lucky).


Regarding the comment in the thread about packages that the installer
added that show up as manual, you can do something like the following
to at least make apt think they were auto:

dpkg-query --show --showformat='${Package} ${Priority}\n' | awk '$2 ==
required {print $1}' > required.txt
sudo apt-mark auto $(apt-mark showmanual | grep -F required.txt) #
apt-mark will prompt, so you don't want to use xargs

Again, the above is untested, so verify first!

You might do the same for other priorities, like standard or
important. If for no other reason than breaking the list of packages
into smaller, digestible chunks that you can focus on. For example,
on my machine:
$ dpkg-query --show --showformat='${Priority}\n' | sort | uniq -c | sort -n
5 extra
29 important
29 standard
33 required
1472 optional

I could probably handle going through those smaller collections to
identify where they came from fairly easily. But that big optional
collection, not so much. For something like that, I might add
${Section} to the --showformat option, and divide them up that way.

Also, as a future project, you might consider creating metapackages to
help organize your installation. Again, for my machine:
$ apt-mark showmanual | wc -l
1
$ apt-mark showauto | wc -l
1563

I have a handful of debian control files that I use (base, desktop,
dev, serviceX, serviceY, machine1, machine2,...). The machine ones
depends on the services they host (NFS, LDAP, VMs), and whether they
need a GUI (desktop), whether I build on them (dev), or play games,
etc. Then each machine, after a base install I do something like:

apt-mark auto $(apt-mark showmanual)
apt install machineN
apt autoremove --purge

Of course, I monitor that autoremove to make sure it doesn't do
anything silly, and if it tries to remove a package I missed, I go add
it to the appropriate control file. My simple little way of doing
this is:

$ cat doit.sh
#!/bin/bash

for v in *.control; do
equivs-build $v > $v.log &
done

echo 'Waiting....'
wait
echo 'Done waiting'

OUTPUT=/srv/deb/packages
rm -rf $OUTPUT
mkdir -p $OUTPUT
cp *.deb $OUTPUT
cd $OUTPUT

dpkg-scanpackages . > Packages
$ cat /etc/apt/sources.list.d/mrc-home.list
deb [trusted=yes] file:/srv/deb/packages ./

And yes, I should do better than the [trusted=yes].

Good luck on your upgrade!
mrc

[Mike Castle]

Oops. The 'grep -v -F' should be 'grep -v -f'. Well, 'grep -v -F -f'
would probably be appropriate as well.

mrc

[Max Nikulin]

On 04/10/2023 09:58, Mike Castle wrote:
> The following UNTESTED commands (ran as a normal user):
>
> (apt-mark showauto ; apt-mark showmanual) > apt-thinks-you-installed.txt
> dpkg-query --show --showformat='${Package}\n' | grep -v -F
> apt-thinks-you-installed.txt > rest.txt
>
> The file "rest.txt" should have a list of packages installed that were
> NOT installed via apt. With any luck, it is small enough to examine
> manually.

My expectation is that rest.txt contains a list of broken or removed,
but not purged packages and an alternative way to get a similar list is

apt list '?config-files | ?broken'

or 'apt list '~c|~b'. I wonder what is the output of

dpkg-query --show --showformat='${Status} ${Package}\n' PKG

for any package from rest.txt that is still installed.

[David Wright]

On Tue 03 Oct 2023 at 19:58:57 (-0700), Mike Castle wrote:
> Some tools I've been using lately are apt-mark and "dpkg-query --show".
>
> The following UNTESTED commands (ran as a normal user):
>
> (apt-mark showauto ; apt-mark showmanual) > apt-thinks-you-installed.txt

> dpkg-query --show --showformat='${Package}\n' | grep -v -F -f
> apt-thinks-you-installed.txt > rest.txt

(I've added the omitted -f.)

> The file "rest.txt" should have a list of packages installed that were
> NOT installed via apt. With any luck, it is small enough to examine
> manually.

I don't think your grep will work correctly. apt-thinks-you-installed.txt
contains patterns, and some of those patterns are very short, for example:
an at bc dc di gv jq mc pv tk acl ant apt bbe cpp ftp git gpg gpm kbd
lz4 mbr mpv mtr pia sed sox tar tcl ucf ufw ure vim w3m xli xxd zip.
These will match packages in the dpkg-query whose names don't actually
appear as complete strings in apt-thinks-you-installed.txt. For example,
pv is in mpv, bbe is in librubberband2. "an" and "at" occur in scores
of package names.

I think you need to enclose each pattern (ie each line of
apt-thinks-you-installed.txt) in the anchors, ^ and $. Ironically,
it would be trivial to add these characters to the output of
dpkg-query, but that's not where they're needed.

Cheers,
David.

[John Crawley]

On 05/10/2023 13:15, David Wright wrote:
> On Tue 03 Oct 2023 at 19:58:57 (-0700), Mike Castle wrote:
>> (apt-mark showauto ; apt-mark showmanual) > apt-thinks-you-installed.txt
>> dpkg-query --show --showformat='${Package}\n' | grep -v -F -f
>> apt-thinks-you-installed.txt > rest.txt
> (I've added the omitted -f.)
>
>> The file "rest.txt" should have a list of packages installed that were
>> NOT installed via apt. With any luck, it is small enough to examine
>> manually.
>
> I don't think your grep will work correctly. apt-thinks-you-installed.txt
> contains patterns, and some of those patterns are very short, for example:
> an at bc dc di gv jq mc pv tk acl ant apt bbe cpp ftp git gpg gpm kbd

> ...

Doesn't the -F option mean grep is treating each line in apt-thinks-you-installed.txt as a fixed string, not as a pattern?

However, running the above commands here results in a short (~10) list of packages none of which I have installed.
dpkg-query --show --showformat='${db:Status-Abbrev} ${Package}\n' outputs a few lines that don't start with "ii" like all the installed packages. I guess they need to be filtered out?

--
John

[Greg Wooledge]

On Thu, Oct 05, 2023 at 03:00:20PM +0900, John Crawley wrote:
> On 05/10/2023 13:15, David Wright wrote:
> > On Tue 03 Oct 2023 at 19:58:57 (-0700), Mike Castle wrote:
> > > (apt-mark showauto ; apt-mark showmanual) > apt-thinks-you-installed.txt
> > > dpkg-query --show --showformat='${Package}\n' | grep -v -F -f
> > > apt-thinks-you-installed.txt > rest.txt
> > (I've added the omitted -f.)
> >
> > > The file "rest.txt" should have a list of packages installed that were
> > > NOT installed via apt. With any luck, it is small enough to examine
> > > manually.
> >
> > I don't think your grep will work correctly. apt-thinks-you-installed.txt
> > contains patterns, and some of those patterns are very short, for example:
> > an at bc dc di gv jq mc pv tk acl ant apt bbe cpp ftp git gpg gpm kbd
> > ...
>
> Doesn't the -F option mean grep is treating each line in apt-thinks-you-installed.txt as a fixed string, not as a pattern?

Correct. You're probably looking for -x instead of adding anchors.