deb822 sources by default for bookworm

[Julian Andres Klode]

Hi all,

I'd like us to move from

/etc/apt/sources.list

to
/etc/apt/sources.list.d/debian.sources

in bookworm.

# deb822 intro

The deb822 format can be shorter and easier to read, to quote the
sources.list manual page:

As an example, the sources for your distribution could look like this in one-line-style format:

deb http://deb.debian.org/debian bullseye main contrib non-free
deb http://security.debian.org bullseye-security main contrib non-free

or like this in deb822 style format:

Types: deb
URIs: http://deb.debian.org/debian
Suites: bullseye
Components: main contrib non-free

Types: deb
URIs: http://security.debian.org
Suites: bullseye-security
Components: main contrib non-free


Now if you mix unstable and testing, you could just have that it one
paragraph.

The big advantage of deb822 sources is that you can embed larger data:

Types: deb
URIs: https://deb.debian.org
Suites: stable
Components: main contrib non-free
Signed-By:
-----BEGIN PGP PUBLIC KEY BLOCK-----
.
mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY
CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk
IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS
dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG
3bHcln8DMpIJVXht78sL
=IE0r
-----END PGP PUBLIC KEY BLOCK-----

Oh yeah, a standalone sources file with embedded key - making third-party
repository management substantially more convenient.

# issues

several software does not support deb822 currently. I plan to work on
adding deb822 support to python-apt in the upcoming months, I don't know
what else is affected.

There is some software "parsing" sources.list on its own, most of that
is better served by `apt-get indextargets` (and for downloading stuff
based on the urls, `apt-helper download-file`, such that it respects
proxies and supports all transports users may use in sources.list)

In terms of setting up system, I guess debootstrap and d-i's apt-setup
module need changes? I admit I do not have a total overview.

# timeline

I do not know the total outcome of this. My hope would be that
we can switch the default in Summer 2022 and see what breaks,
although I don't know who's going to install from testing
d-i :D

docker images probably can move earlier as they don't have
as much interactive users that use tools that might be broken
(e.g. software-properties). Possibly others, there's no need
to roll out in multiple places at the same time, as long as
we converge by freeze time.

I invite people to test this out already on their own systems,
I know several people do; and extrepo also builds deb822 sources
files, but several people I guess do not know about it yet. Please
test and report bugs.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en

[Shengjing Zhu]

On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode <j...@debian.org> wrote:
>
> Hi all,
>
> I'd like us to move from
>
> /etc/apt/sources.list
>
> to
> /etc/apt/sources.list.d/debian.sources
>

While it's really a nice feature for the third-party repository, I
don't see the benefits to change the default one, especially the path.
I had to admit that I have countless scripts which run `sed
/etc/apt/souces.list`, to change the default mirror, as well as in the
Dockerfile.

--
Shengjing Zhu

[Holger Levsen]

Hi Julian,

this sounds like a nice and useful plan and feature(s), thank you!
just one question:

On Wed, Nov 03, 2021 at 04:45:15PM +0100, Julian Andres Klode wrote:
> I'd like us to move from
> /etc/apt/sources.list
> to
> /etc/apt/sources.list.d/debian.sources

[...]
> #timeline

You didn't say so explicitly, but do you plan to support old style
/etc/apt/sources.list until forever? ;) Or do you envision automatic
migration of that file? Or?


--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

Only change is constant.

[Julian Andres Klode]

On Wed, Nov 03, 2021 at 04:23:52PM +0000, Holger Levsen wrote:
> Hi Julian,
>
> this sounds like a nice and useful plan and feature(s), thank you!
> just one question:
>
> On Wed, Nov 03, 2021 at 04:45:15PM +0100, Julian Andres Klode wrote:
> > I'd like us to move from
> > /etc/apt/sources.list
> > to
> > /etc/apt/sources.list.d/debian.sources
> [...]
> > #timeline
>
> You didn't say so explicitly, but do you plan to support old style
> /etc/apt/sources.list until forever? ;) Or do you envision automatic
> migration of that file? Or?

I don't know, to be honest, have not thought about it yet.

With APT's 5 year interface stability and major version bump,
The first time we could remove support for old releases would be
trixie/apt 3.0 in 2025 (that schedule just happens because I don't
want to release .10 versions, but keep the 2nd component single
digit :D)

I think an automatic migration might be to painful what with all the
juju and ansible and saltstack (I feel like it'd be nice to have
those tools migrate config to new formats).

Of course, once everyone and their dog has migrated, I might feel
different and complain about legacy sources.list and deprecate
them.

So presumably, we'd have both as supported in bookworm, sources.list
deprecated for trixie, and then removed in 2030 in apt 4.0, but that's
still 8 years to go.

(all dates are 1 year after the change lands in unstable, as the
changes land in odd numbers earlier in the cycle)

[Holger Levsen]

On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote:
> I don't know, to be honest, have not thought about it yet.

many thanks for your verbose reply! /me likes this timeline.

> I think an automatic migration might be to painful what with all the
> juju and ansible and saltstack (I feel like it'd be nice to have
> those tools migrate config to new formats).

I guess it would be nice if those tools (and users not using those
tools) could run one standard tool doing the job :)

--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

This too shall pass.

[Paul Gevers]

Hi Julian,

On 03-11-2021 16:45, Julian Andres Klode wrote:
> There is some software "parsing" sources.list on its own, most of that
> is better served by `apt-get indextargets` (and for downloading stuff
> based on the urls, `apt-helper download-file`, such that it respects
> proxies and supports all transports users may use in sources.list)

Like autopkgtest. When I was working on it to support Debian's migration
testing, I looked at python-apt and because that didn't support it,
stopped thinking. With indextargets and download-file I guess we could
work on it again. When were those introduced? Ubuntu needs it on old
releases so before autopkgtest can change it, we'd need support for a while.

Paul

[Julian Andres Klode]

download-file gained hash checking support in 0.9.16, did not find when
it was introduced.

indextargets has been around since 1.1.

(I'm not subscribed, please Cc me, I don't check archives/nntp often)

[Thomas Goirand]

On 11/3/21 5:59 PM, Holger Levsen wrote:
> On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote:
>> I don't know, to be honest, have not thought about it yet.
>
> many thanks for your verbose reply! /me likes this timeline.
>
>> I think an automatic migration might be to painful what with all the
>> juju and ansible and saltstack (I feel like it'd be nice to have
>> those tools migrate config to new formats).
>
> I guess it would be nice if those tools (and users not using those
> tools) could run one standard tool doing the job :)

This feels like a very nicely thought plans, which I 100% support. I
especially like the timeline for deprecating and removal, which gives
enough time for everyone to support the new format.

So I'm joining to say thank you to Julian.

Cheers,

Thomas Goirand (zigo)

[Colin Watson]

On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote:
> On Wed, Nov 03, 2021 at 04:23:52PM +0000, Holger Levsen wrote:
> > You didn't say so explicitly, but do you plan to support old style
> > /etc/apt/sources.list until forever? ;) Or do you envision automatic
> > migration of that file? Or?
>
> I don't know, to be honest, have not thought about it yet.
>
> With APT's 5 year interface stability and major version bump,
> The first time we could remove support for old releases would be
> trixie/apt 3.0 in 2025 (that schedule just happens because I don't
> want to release .10 versions, but keep the 2nd component single
> digit :D)
>
> I think an automatic migration might be to painful what with all the
> juju and ansible and saltstack (I feel like it'd be nice to have
> those tools migrate config to new formats).
>
> Of course, once everyone and their dog has migrated, I might feel
> different and complain about legacy sources.list and deprecate
> them.

I think it would be a mistake to ever drop support for the older format.
Instructions for using it are scattered all over the place, and there'll
be an enormous long tail of things to update, not to mention things like
old forum posts that can never really be updated.

I agree we might feel different in ten years' time!

--
Colin Watson (he/him) [cjwa...@debian.org]

[David Kalnischkies]

On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote:

`apt-get indextargets` is from 2015 and a part of the acquire-additional-
files feature used mainly by apt-file and appstream to have apt download
files it isn't using itself, so those tools don't have to implement it.

The job of indextargets is it mostly to give access to metadata (and
crucially filenames) for those previously configured and hopefully now
downloaded files. apt-file e.g. asks for the Contents files in this way
to avoid exposing file naming logic and location to other tools.

So, for the filenames of all (downloaded) Packages files:
apt-get indextargets --format '$(FILENAME)' 'Identifier: Packages'
(the default output is deb822 stanzas you could grep with more powerful
tools than the simple inbuilt line-based filter)

Note that you either have to implement opening compressed files yourself
or use `/usr/lib/apt/apt-helper cat-file`.


That was historically the most common reason to fiddle with sources.list
parsing, hence Julian referring to it, but this seems not what autpkgtest
is aiming for. On a casual look (well, grep) I see only:

* lib/adt_testbed.py apt-pocket codepath seems to want to construct new
sources.list entries based on existing ones. That should be possible
with some indextargets busywork in general, but I am not completely
sure what is going on here and the gymnastics should be similar to…

* setup-commands/setup-testbed tries to find the mirror (and release)
used for target distribution based on your current system. I am
a bit surprised that works actually…

Anyway, the later could perhaps be implement with:
apt-get indextargets --format '$(ORIGIN)|$(REPO_URI)' | sort -u | \
grep -i -e "^$(. /etc/os-release; echo "$ID")" | cut -d'|' -f 2

(That is one line only for posterity – as you see, I am trying to fix
the too general search by checking against Origin as defined by the
Release file of a repository, but that would need work still to
eliminate same-origin-but-different-repo cases)


Parsing of the sources files is not really indextargets job through, so
it might not always work for that task: It e.g. doesn't work if the data
files are not on disk which might or might not be okay for you (there is
'guess' mode, but that of course has no metadata extracted from the
Release files – the Origin I was using above).

The apt family doesn't really have a publicly exposed way of reasoning
about sources.list (or .sources) files and I am not quite sure it really
can as subtil differences between repositories make it hard to give them
all a common interface which makes sense. (I will probably be proven
wrong by Julian though.)


Like, for example, if stable is in the sources, make sure there
is also updates and security there and/or add them. What for Debian are
three distinct repositories might for others very well be components.

Assuming you even know which line refers to Debian: I was using Origin
above for this task as we can't really guess based on the URI. And even
then… that logic above finds the tor+mirror+file source I am using,
that won't work for autopkgtest, but I am special and this just
a default fallback, so I might be thinking way too much about it…


Anyway, if you have specific needs/questions feel free to ask on deity@
or #debian-apt. I am sure we will work something out even if in this
case it might very well be new code nobody really uses for years (as is
common in apt land – backward compat be damned 😉).

Best regards

David Kalnischkies

[Julian Andres Klode]

On Thu, Nov 04, 2021 at 12:13:48AM +0800, Shengjing Zhu wrote:
> On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode <j...@debian.org> wrote:
> >
> > Hi all,
> >
> > I'd like us to move from
> >
> > /etc/apt/sources.list
> >
> > to
> > /etc/apt/sources.list.d/debian.sources
> >
>
> While it's really a nice feature for the third-party repository, I
> don't see the benefits to change the default one, especially the path.
> I had to admit that I have countless scripts which run `sed
> /etc/apt/souces.list`, to change the default mirror, as well as in the
> Dockerfile.

There's a technical limitation in that we get the format from the file
extension. It's a bit annoying.

Also this makes multi-origin downstream distros cleaner, so you can
have

/etc/apt/sources.list.d/debian.sources
/etc/apt/sources.list.d/downstream.sources

instead of worrying about either creating a combined sources.list or
dropping just your downstream bits into sources.list.d

We'll have to see how much trouble that causes, and it might work
to make sources.list a symlink to sources.list.d/debian.sources,
and have apt inspect readlink() to get the "real" file extension
for sources.list

[Julian Andres Klode]

We could render what we passed as a deb822 format in apt{,-get}
sources list (remind me to add subcommand support to the parser).

Possibly with some normalization in there. Presumably if you have
a paragraph "Suites: foo bar Components: x y" you end up with 2
paragraphs in the output, as IIRC we group our sources entries
by Release file.

I think that provides a reasonable top-level interface. You could
of course then also normalize this further such that all fields only
have one value, and give them all IDs and then you can apt source
remove <id> or stuff, idk.

Will have to see how I implement the deb822 parsing and editing
on the python-apt side.

[David Kalnischkies]

On Fri, Nov 05, 2021 at 06:16:10PM +0100, Julian Andres Klode wrote:
> On Thu, Nov 04, 2021 at 12:13:48AM +0800, Shengjing Zhu wrote:
> > On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode <j...@debian.org> wrote:
> > >
> > > Hi all,
> > >
> > > I'd like us to move from
> > >
> > > /etc/apt/sources.list
> > >
> > > to
> > > /etc/apt/sources.list.d/debian.sources
> > >
> >
> > While it's really a nice feature for the third-party repository, I
> > don't see the benefits to change the default one, especially the path.
> > I had to admit that I have countless scripts which run `sed
> > /etc/apt/souces.list`, to change the default mirror, as well as in the
> > Dockerfile.

(fwiw you could use a 'mirror+file' entry (see man apt-transport-mirror)
specifying your preferred mirror of the day. As a bonus that will reuse
the files of the old mirror instead of discarding them blindly)

> There's a technical limitation in that we get the format from the file
> extension. It's a bit annoying.

It was shortly the case that .list files could contain both formats, but
I opted to change that as of course countless things poking into those
files broke left and right.

cups used to detect if it ran on a Debian-like system by checking the
sources.list file for deb entries… I doubt it does nowadays as there are
countless better options now, I just mention it as trivial example of
the type of unexpected breakage which is to be expected…


Best regards

David Kalnischkies