[PATCH] Dockerfile: add python3-venv to container

[ross....@arm.com]

From: Ross Burton <ross....@arm.com>

If you try and do the right thing and use a venv to run other Python
applications in a kas container, it fails:

$ python3 -m venv myvenv
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt-get install python3-venv

Signed-off-by: Ross Burton <ross....@arm.com>
---
Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Dockerfile b/Dockerfile
index 0e79cb5..a2ea2ed 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -31,7 +31,7 @@ RUN apt-get update && \
localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 && \
apt-get install --no-install-recommends -y \
python3-pip python3-setuptools python3-wheel python3-yaml python3-distro python3-jsonschema \
- python3-newt python3-colorlog python3-kconfiglib \
+ python3-newt python3-colorlog python3-kconfiglib python3-venv \
gosu lsb-release file vim less procps tree tar bzip2 zstd pigz lz4 unzip tmux libncurses-dev \
git-lfs mercurial iproute2 ssh-client telnet curl rsync gnupg awscli sudo \
socat bash-completion && \
--
2.34.1

[Jan Kiszka]

On 04.07.23 17:22, ross....@arm.com wrote:
> From: Ross Burton <ross....@arm.com>
>
> If you try and do the right thing and use a venv to run other Python
> applications in a kas container, it fails:
>
> $ python3 -m venv myvenv
> The virtual environment was not created successfully because ensurepip is not
> available. On Debian/Ubuntu systems, you need to install the python3-venv
> package using the following command.
> apt-get install python3-venv

OK... Is that a regression? Of the 4.0 container? And which builds are
triggering this? Or is it a manual thing via kas shell?

Jan

>
> Signed-off-by: Ross Burton <ross....@arm.com>
> ---
> Dockerfile | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Dockerfile b/Dockerfile
> index 0e79cb5..a2ea2ed 100644
> --- a/Dockerfile
> +++ b/Dockerfile
> @@ -31,7 +31,7 @@ RUN apt-get update && \
> localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 && \
> apt-get install --no-install-recommends -y \
> python3-pip python3-setuptools python3-wheel python3-yaml python3-distro python3-jsonschema \
> - python3-newt python3-colorlog python3-kconfiglib \
> + python3-newt python3-colorlog python3-kconfiglib python3-venv \
> gosu lsb-release file vim less procps tree tar bzip2 zstd pigz lz4 unzip tmux libncurses-dev \
> git-lfs mercurial iproute2 ssh-client telnet curl rsync gnupg awscli sudo \
> socat bash-completion && \

--

Siemens AG, Technology
Competence Center Embedded Linux

[Ross Burton]

Hi Jan,

On 4 Jul 2023, at 16:36, Jan Kiszka <jan.k...@siemens.com> wrote:
>
> On 04.07.23 17:22, ross....@arm.com wrote:
>> From: Ross Burton <ross....@arm.com>
>>
>> If you try and do the right thing and use a venv to run other Python
>> applications in a kas container, it fails:
>>
>> $ python3 -m venv myvenv
>> The virtual environment was not created successfully because ensurepip is not
>> available. On Debian/Ubuntu systems, you need to install the python3-venv
>> package using the following command.
>> apt-get install python3-venv
>
> OK... Is that a regression? Of the 4.0 container? And which builds are
> triggering this? Or is it a manual thing via kas shell?

It’s a combination of factors.

venvs don’t work in the 3.3 container either, as python3-venv isn’t present in that either.

The change in behaviour, where ‘sudo pip3 install package’ refuses to work, is a Debian 12 change. You either apt install to the system, or pip install to a virtual environment. Our CI was ‘sudo pip install”ing to build documentation in the same container and that was failing with the move to the 4.0 container.

Ross

[Jan Kiszka]

Ok, so another workaround would be adding --break-system-packages, like
we do in our Dockerfile as well.

I don't mind adding venv to the base container as well if that serves
more or less common use cases. OTOH, you are already installing stuff to
your container instance, just via pip rather than via apt. What is the
point of having 1 out of n pieces pre-installed? And you do need to
adjust your CI scripts as well by switching to venv.

Jan

[Ross Burton]

On 4 Jul 2023, at 16:50, Jan Kiszka <jan.k...@siemens.com> wrote:
>> The change in behaviour, where ‘sudo pip3 install package’ refuses to work, is a Debian 12 change. You either apt install to the system, or pip install to a virtual environment. Our CI was ‘sudo pip install”ing to build documentation in the same container and that was failing with the move to the 4.0 container.
>
> Ok, so another workaround would be adding --break-system-packages, like
> we do in our Dockerfile as well.

Right, and that seems… suboptimal.

> I don't mind adding venv to the base container as well if that serves
> more or less common use cases. OTOH, you are already installing stuff to
> your container instance, just via pip rather than via apt. What is the
> point of having 1 out of n pieces pre-installed? And you do need to
> adjust your CI scripts as well by switching to venv.

I guess the point is that with this patch, venv works out of the box.

Ross

[Jan Kiszka]

On 04.07.23 18:20, Ross Burton wrote:
> On 4 Jul 2023, at 16:50, Jan Kiszka <jan.k...@siemens.com> wrote:
>>> The change in behaviour, where ‘sudo pip3 install package’ refuses to work, is a Debian 12 change. You either apt install to the system, or pip install to a virtual environment. Our CI was ‘sudo pip install”ing to build documentation in the same container and that was failing with the move to the 4.0 container.
>>
>> Ok, so another workaround would be adding --break-system-packages, like
>> we do in our Dockerfile as well.
>
> Right, and that seems… suboptimal.

Well, the proper one would be building and installing a Debian package
of kas. Doable, just a bit overkill for the container image.

>
>> I don't mind adding venv to the base container as well if that serves
>> more or less common use cases. OTOH, you are already installing stuff to
>> your container instance, just via pip rather than via apt. What is the
>> point of having 1 out of n pieces pre-installed? And you do need to
>> adjust your CI scripts as well by switching to venv.
>
> I guess the point is that with this patch, venv works out of the box.

Right, but that's already it. No feature works on top that does not
require manipulation of the image in the first place. I'm all for adding
value to the image that covers build use cases (check-mark for yours)
and avoids having to touch it otherwise (no check-mark for this).

[Moessbauer Felix]

On Tue, 2023-07-04 at 19:03 +0200, 'Jan Kiszka' via kas-devel wrote:
> On 04.07.23 18:20, Ross Burton wrote:
> > On 4 Jul 2023, at 16:50, Jan Kiszka <jan.k...@siemens.com> wrote:
> > > > The change in behaviour, where ‘sudo pip3 install package’
> > > > refuses to work, is a Debian 12 change. You either apt install
> > > > to the system, or pip install to a virtual environment.  Our CI
> > > > was ‘sudo pip install”ing to build documentation in the same
> > > > container and that was failing with the move to the 4.0
> > > > container.
> > >
> > > Ok, so another workaround would be adding --break-system-
> > > packages, like
> > > we do in our Dockerfile as well.
> >
> > Right, and that seems… suboptimal.
>
> Well, the proper one would be building and installing a Debian
> package
> of kas. Doable, just a bit overkill for the container image.

I anyways thought about packaging kas for debian. In bookworm we
already have all required dependencies. If we use a multi-stage build,
it is rather straight forward to build the package and then install
that into the final container.

In addition, this would avoid the python build dependencies in the
container and make it a bit smaller (e.g. pip, setuptools, wheel).

Felix

[Jan Kiszka]

We could likely use what Debian has:
https://packages.debian.org/trixie/kas