Bug#909299: simple-cdd: should not fail when "extrafiles" doesn't exist

[Raphaël Hertzog]

Package: simple-cdd
Version: 0.6.6
Severity: normal

Version 0.6.5 introduced a checksum check based on the data from
the "extrafiles" file at the root of the mirror. Now when that
file doesn't exist, simple-cdd fails with a stacktrace and is
unable to build any image.

Arguably, the lack of this file is not a showstopper, it should
just generate a warning... and effectively there are many custom
Debian repositories without this file and you still want to be able
to run simple-cdd on them.

In my specific case, the Debian mirror is created with "debmirror"
and this tool doesn't include that file. But I could also mention the
case of many derivatives that just use reprepro.

Also it would be nice if simple-cdd documented somewhere its requirements for
the mirror and repositories that it can use.

FTR here's the stacktrace:
2018-09-18 14:36:26,005 DEBUG Building local Debian mirror for debian-cd...
2018-09-18 14:36:26,007 DEBUG downloading: .../tmp/mirror/extrafiles
Traceback (most recent call last):
File "/usr/bin/build-simple-cdd", line 658, in <module>
scdd.build_mirror()
File "/usr/bin/build-simple-cdd", line 270, in build_mirror
self.run_tool("mirror", tool)
File "/usr/bin/build-simple-cdd", line 367, in run_tool
tool.run()
File "/usr/lib/python3/dist-packages/simple_cdd/tools/mirror_wget.py", line 64, in
run
_download(download_extrafiles_file, extrafiles_file_inlinesig)
File "/usr/lib/python3/dist-packages/simple_cdd/tools/mirror_wget.py", line 55, in
_download
request.urlretrieve(url, filename=output)
File "/usr/lib/python3.5/urllib/request.py", line 188, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found


-- System Information:
Debian Release: buster/sid
APT prefers oldoldstable
APT policy: (500, 'oldoldstable'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.18.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages simple-cdd depends on:
ii dctrl-tools 2.24-2+b1
ii debian-cd 3.1.21
ii lsb-release 9.20170808
ii python3 3.6.6-1
ii python3-simple-cdd 0.6.6
ii reprepro 5.2.0-1
ii rsync 3.1.2-2.2
ii wget 1.19.5-2

Versions of packages simple-cdd recommends:
ii dose-distcheck 5.0.1-11+b1

Versions of packages simple-cdd suggests:
ii qemu-kvm 1:2.12+dfsg-3

-- no debconf information

[Vagrant Cascadian]

On 2018-09-21, Raphaël Hertzog wrote:
> Version 0.6.5 introduced a checksum check based on the data from
> the "extrafiles" file at the root of the mirror. Now when that
> file doesn't exist, simple-cdd fails with a stacktrace and is
> unable to build any image.
>
> Arguably, the lack of this file is not a showstopper, it should
> just generate a warning... and effectively there are many custom
> Debian repositories without this file and you still want to be able
> to run simple-cdd on them.

Before simple-cdd used extrafiles, it would blindly download
debian-installer files from the mirror with no verification, and
recursively get entire directory trees...

Using extrafiles enables a signed trust path to checksums of various
non-package files, and conveniently lists the files available to
download on the archive as an added bonus.

So it's non-trivial to add support for arbitrary files in arbitrary
directories in a secure manner...

> In my specific case, the Debian mirror is created with "debmirror"
> and this tool doesn't include that file. But I could also mention the
> case of many derivatives that just use reprepro.

Maybe these other tools could add support for extrafiles?

It's unfortunate that it may not work in all environments, though
simple-cdd has always targeted building images with files from
debian.org, and not arbitrary locations.

A patch to enable support without extrafiles would, of course, be
considered if it didn't risk degrading the trust path by default.

> Also it would be nice if simple-cdd documented somewhere its requirements for
> the mirror and repositories that it can use.

That's surely doable.

Yeah, simple-cdd should at least handle this rather than spitting out a
backtrace.


Thanks for the report, sorry I don't have better news for this issue!


live well,
vagrant

[Raphael Hertzog]

Hi,

On Mon, 27 May 2019, Vagrant Cascadian wrote:
> So it's non-trivial to add support for arbitrary files in arbitrary
> directories in a secure manner...

I would argue that the security aspect here is somewhat irrelevant.
simple-cdd is run against a repository/mirror that is selected by the
user. If he selects a malicious mirror, the malicious mirror can do
much more harm... no matter whether this file exists or not.

> > In my specific case, the Debian mirror is created with "debmirror"
> > and this tool doesn't include that file. But I could also mention the
> > case of many derivatives that just use reprepro.
>
> Maybe these other tools could add support for extrafiles?

Maybe, but it's unlikely. Those tools are meant to create/
mirror APT repositories and arguably this extrafile is just
not part of any concept of an APT repository.

To me it's really clear that it's simple-cdd that should
have the required flexibility.

> It's unfortunate that it may not work in all environments, though
> simple-cdd has always targeted building images with files from
> debian.org, and not arbitrary locations.

When you sell yourself as a tool to create "Custom Debian Distribution",
IMO you should support being built against custom debian mirror.

Due to the flexibility of simple-cdd, it has always been used by
private derivatives and the like, it would be sad to lose those users.

> A patch to enable support without extrafiles would, of course, be
> considered if it didn't risk degrading the trust path by default.

I'm sorry, I don't have the time for this. Do you want to tag this bug
help then?

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

[Vagrant Cascadian]

Control: tags 909299 help

On 2019-05-28, Raphael Hertzog wrote:
> On Mon, 27 May 2019, Vagrant Cascadian wrote:
>> So it's non-trivial to add support for arbitrary files in arbitrary
>> directories in a secure manner...
>
> I would argue that the security aspect here is somewhat irrelevant.
> simple-cdd is run against a repository/mirror that is selected by the
> user. If he selects a malicious mirror, the malicious mirror can do
> much more harm... no matter whether this file exists or not.

Depending on it does at least prevent man-in-the-middle attacks against
a valid mirror, which seems quite security relevent.

>> > In my specific case, the Debian mirror is created with "debmirror"
>> > and this tool doesn't include that file. But I could also mention the
>> > case of many derivatives that just use reprepro.
>>
>> Maybe these other tools could add support for extrafiles?
>
> Maybe, but it's unlikely. Those tools are meant to create/
> mirror APT repositories and arguably this extrafile is just
> not part of any concept of an APT repository.

The extrafiles file documents the parts of the repository which are not
part of the APT repository but are present in the Debian mirror. These
extra files include things such as debian-installer images,
documentation, and tools which are not shipped inside the repository in
any way that APT can make use of them.

I'm not aware of any other clean way to discover what files the mirror
makes available, and the available files change names and locations from
time to time... so it would be fragile to hard-code them.


If you provide your own debian-installer image (using
"custom_installer"), and use various debian-cd OMIT_* options such as:

export OMIT_MANUAL=1
export OMIT_DOC_TOOLS=1

Then I believe simple-cdd won't even attempt to use extrafiles.

>> It's unfortunate that it may not work in all environments, though
>> simple-cdd has always targeted building images with files from
>> debian.org, and not arbitrary locations.
>
> When you sell yourself as a tool to create "Custom Debian Distribution",
> IMO you should support being built against custom debian mirror.

It was named before the term Debian Pure Blends existed (not that I
think the new name clears up the confusion at all), and really was
intended as a tool to make quick selections of things within Debian,
with small and minor customizations with the goal of being able to
integrate them into Debian eventually. Maybe it's been primarily useful
outside of accomplishing those aims... and that's fine!

> Due to the flexibility of simple-cdd, it has always been used by
> private derivatives and the like, it would be sad to lose those users.

I'm glad to hear it's been useful!

I have unfortunately not had much time to invest in it over the years
beyond just keeping up with changes in Debian, debian-installer and
debian-cd. That said, I've kept it going this past 14.5 years...

Would love it if more people could help improve it to meet whatever
shortcomings it may have. The python rewrite, was one such great
improvement, despite a few flaws.

>> A patch to enable support without extrafiles would, of course, be
>> considered if it didn't risk degrading the trust path by default.
>
> I'm sorry, I don't have the time for this. Do you want to tag this bug
> help then?

Me neither, tagging as help.


live well,
vagrant