Installing docker images with swupdate

[Aldenburg, Felix]

Hello,

we would like to install docker images with swupdate.

 

These images are later used by our application. (You can think of it as a app manager similar to ones on smartphones, but using docker images/container.)

These docker images may potential be quite large size, and we have limited disk space so we’d like to import them directly into the docker engine without having to store them temporarily.

 

As a POC, we did some testing with a custom lua and c handler, where we streamed the docker image from the .swu file into the docker shell cli tool.

 

lua_docker_handler.lua

 

require ("swupdate")

install_docker_image = function(image)           

    swupdate.trace("Lua docker handler started")

    local dockerhandle = io.popen("docker load", "w")

    err, msg = image:read(function(data)dockerhandle:write(data) end)

    if err ~= 0 then

        swupdate.error(string.format("Error: %s", msg))

        dockerhandle:close()

        return 1

    end

    dockerhandle:close()

    swupdate.trace("Lua docker handler end")

    return 0

end

swupdate.register_handler("lua_docker_handler", install_docker_image, swupdate.HANDLER_MASK.IMAGE_HANDLER)

 

 

While our POC works, we are wondering if there are better ways to utilize swupdate to action this task.

Or perhaps this totally out of scope for swupdate?

Would a docker image import handler be an interesting asset for the swupdate project?

 

Looking forward to receiving your feedback and thoughts on the topic.

 

Best regards
Felix Aldenburg

 

---

Kommanditgesellschaft - Sitz: Detmold - Amtsgericht Lemgo HRA 2790 -
Komplementärin: Weidmüller Interface Führungsgesellschaft mbH -
Sitz: Detmold - Amtsgericht Lemgo HRB 3924;
Geschäftsführer: Dr. Timo Berger, Volker Bibelhausen, André Sombecki;
USt-ID-Nr. DE124599660

[Stefano Babic]

Hallo Felix,

On 24.08.22 14:55, Aldenburg, Felix wrote:
> Hello,
>
> we would like to install docker images with swupdate.
>

Ok

> These images are later used by our application. (You can think of it as
> a app manager similar to ones on smartphones, but using docker
> images/container.)

This is already a known pattern, I have done this, too.

>
> These docker images may potential be quite large size, and we have
> limited disk space so we’d like to import them directly into the docker
> engine without having to store them temporarily.

This seems difficult - I switched to podman as replacement for docker,
more or less due to the critics about docker (and I had several cases
where docker daemon did not run). However, it seems to me (and as far as
I remember, this what reported by someone else in the list), that
"podman load" first copies the tarball before creating the image, and
this vanifies the streaming feature in SWUpdate. But it was not so
critical in those projects of mine, and I have not investigated further.
Just remark this if you have different experiences.

>
> As a POC, we did some testing with a custom lua and c handler, where we
> streamed the docker image from the .swu file into the docker shell cli tool.
>
> lua_docker_handler.lua
>
> require ("swupdate")
>
> install_docker_image = function(image)
>
>     swupdate.trace("Lua docker handler started")
>
>     local dockerhandle = io.popen("docker load", "w")
>
>     err, msg = image:read(function(data)dockerhandle:write(data) end)
>
>     if err ~= 0 then
>
>         swupdate.error(string.format("Error: %s", msg))
>
>         dockerhandle:close()
>
>         return 1
>
>     end
>
>     dockerhandle:close()
>
>     swupdate.trace("Lua docker handler end")
>
>     return 0
>
> end
>
> swupdate.register_handler("lua_docker_handler", install_docker_image,
> swupdate.HANDLER_MASK.IMAGE_HANDLER)
>
> While our POC works, we are wondering if there are better ways to
> utilize swupdate to action this task.

IMHO the tarball is copied temporarily in the example above - are you
sure there is no additional copy ? Have you monitored /tmp and
/tmp/datadst during the update ? I am expecting that the tarball is
saved temporarily.

>
> Or perhaps this totally out of scope for swupdate?

No, it is not. It can be very interesting to add it to SWUpdate. I added
up now containers' images in some projects with a Lua / shell script,
but there was not strict requirement about space and customer won't
finance a "container handler".

>
> Would a docker image import handler be an interesting asset for the
> swupdate project?

Definetly, supporting both docker and podman. The REST-API is compatible
and the only difference is that a podman service should be started
before the update - my idea was to start the service from the handler
when a container image must be installed.

>
> Looking forward to receiving your feedback and thoughts on the topic.

Best regards,
Stefano Babic

>
> Best regards
> Felix Aldenburg
>
> ------------------------------------------------------------------------

> Kommanditgesellschaft - Sitz: Detmold - Amtsgericht Lemgo HRA 2790 -
> Komplementärin: Weidmüller Interface Führungsgesellschaft mbH -
> Sitz: Detmold - Amtsgericht Lemgo HRB 3924;
> Geschäftsführer: Dr. Timo Berger, Volker Bibelhausen, André Sombecki;
> USt-ID-Nr. DE124599660
>

> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/DU0PR08MB763851784BF8BA1523CC8A13F0739%40DU0PR08MB7638.eurprd08.prod.outlook.com
> <https://groups.google.com/d/msgid/swupdate/DU0PR08MB763851784BF8BA1523CC8A13F0739%40DU0PR08MB7638.eurprd08.prod.outlook.com?utm_medium=email&utm_source=footer>.


--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, 82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[Christian Storm]

Hi,

Well, this topic is a rat hole I have discussed with my colleague
Silvano (CC) on several occasions ― and yet again ;)

Using $ <container_suite> import or equivalents is a very high level
frontend/interface to the process of storing the container image into
the <container_suite>'s local storage, whose (internal) layout should be
and is opaque at this level. However, not so if you want to avoid
multiple temporary copies while the import process...

The main problem it boils down to is the choice of the tar[+gzip] format
as standard back then in the Docker days that still haunts us today as
it is not particularly well suited for this purpose: For example, tar is
not stable in sorting and tar.gzip cannot be seek()'d.
Hence, special libraries/tools making the .tar files stable and thus
reproducible are stuffed in the process to work around this problem ―
which means you cannot use your standard flavor of tar. Otherwise, two
content-wise identical .tar files would have different digests which is
a problem if your <container_suite>'s local storage is laid out content-
addressable.
For .tar.gz files/layers you need two passes to check the digest and
whether it's already in local storage and then import it if not ― or you
need a temporary copy. This is a problem when wanting to SWUpdate in
streaming mode...
This blog post nicely summarizes this and more in greater details:
https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

So, for me, one way forward to work around the multiple temporary copies
problem would be to dissect/pre-process+augment the container image
(layers) into palatable and streamable .swu "sections", e.g., via
swugenerator [1], which are then imported into the <container_suite>'s
local storage with the help of a "<container_suite> handler" leveraging
the <container_suite>'s low(er)-level plumbing tools such as [2]. Still,
there are some strings attached such as properly handling white-outs and
this is specific to a <container_suite>'s local storage idea.
For non-layered/full container images this process gets easier, then
however losing the layer composability and hence wasting disk space...


Kind regards,
Christian

[1] https://github.com/sbabic/swugenerator
[2] https://github.com/containers/storage

--
Dr. Christian Storm
Siemens AG, Technology, T CED SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany

[Stefano Babic]

Hi Christian,

Correct - and handling the internal of a container like docker / podman
is outside SWUpdate responsabilities. It becomes complicated, too,
because each solution (podman, docker, but even systemd-nspawnd) have
different features and way to go.

>
> The main problem it boils down to is the choice of the tar[+gzip] format
> as standard back then in the Docker days that still haunts us today as
> it is not particularly well suited for this purpose: For example, tar is
> not stable in sorting and tar.gzip cannot be seek()'d.
> Hence, special libraries/tools making the .tar files stable and thus
> reproducible are stuffed in the process to work around this problem ―
> which means you cannot use your standard flavor of tar. Otherwise, two
> content-wise identical .tar files would have different digests which is
> a problem if your <container_suite>'s local storage is laid out content-
> addressable.

Sure

> For .tar.gz files/layers you need two passes to check the digest and
> whether it's already in local storage and then import it if not ― or you
> need a temporary copy. This is a problem when wanting to SWUpdate in
> streaming mode...
> This blog post nicely summarizes this and more in greater details:
> https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

Thanks for link, this is interesting. But this explains the issues with
tarballs, and the reason to switch to other formats like OCI. Let's say
we have already switched to OCI. And yes, there are still
incompatibilities, because the process is not alwqays straightforward.
In some cases, container is created with a different SDK / Docker
Version and must be imported by the device with another container
management, like podman. On the paper, they are full compatible, in the
real world is not always true.

But even switching to OCI does not solves the space issue, that is how
to stream to device avoiding temporary copy. From SWUpdate, everythin is
still streamed and then is passed to the specific container app, and
this should be able to install it without copying.

>
> So, for me, one way forward to work around the multiple temporary copies
> problem would be to dissect/pre-process+augment the container image
> (layers) into palatable and streamable .swu "sections", e.g., via
> swugenerator [1], which are then imported into the <container_suite>'s
> local storage with the help of a "<container_suite> handler" leveraging
> the <container_suite>'s low(er)-level plumbing tools such as [2]. Still,
> there are some strings attached such as properly handling white-outs and
> this is specific to a <container_suite>'s local storage idea.

Ok, so to be streamed, we need to solve dependencies. If an image
depends on layers, these layers should be streamed first. How can we
solve or how do you imagine to split up the container to be added in
several SWU sections ? And what about the order in the SWU ? Do we need
some kind of order then ?

> For non-layered/full container images this process gets easier, then
> however losing the layer composability and hence wasting disk space...
>

Best regards,
Stefano

>
> Kind regards,
> Christian
>
> [1] https://github.com/sbabic/swugenerator
> [2] https://github.com/containers/storage
>


--

[Christian Storm]

Hi Stefano,

Well, the order in which the layers are composed is defined by the
container "image" built off of them. So, I *guess* it's not important.
An idea would be to export each layer into an .swu's image {} section
that is handled by a special handler. So you have a list of layers in
your .swu file. This is then processed one-by-one by the plumbing tool
of the respective <container_suite> so not to have temporary copies.
Then, you define a container "image" out of them, which could as well be
shipped in the .swu handled by a different handler.
However, I suspect you need to adapt those plumbing tools to suit your
input pipeline (e.g., respect stdin for streaming, telling it the SHA
to use for the layer, ...).

When not using a <container_suite> such as Docker, podman, ... but
instead a container runtime directly such as runC, systemd-nspawn, ...
then you're not into this problems as you can define the storage and
delivery as long as you end up in a format that's digestible by the
runtime....


Kind regards,
Christian