Patching local files before they're added into the SWU cpio archive

[Dimitrios Bouras]

Hello,

Currently it is possible to patch ("on the fly", e.g. by editing in place using "sed") file sw-description just before it's added into the SWU cpio archive because do_swupdate() copies it from $WORKDIR to $S (= ${WORKDIR}/${PN}).

Unfortunately the same is not true for any other local file because swupdate_add_src_uri() (called by do_swupdate) copies everything else directly from SRC_URI, instead of from $WORKDIR (contrary to usual practice).

I understand there are advantages in doing things this way. I'm looking for an elegant "work-around" to patch local source files before they are copied from SRC_URI into $S (in preparation for the call to cpio for producing the .swu file).

A bit of background for context: the files I'm interested in patching via "sed" are in fact scripts that have fall-back defaults saved in variables. It is desirable to generate (and "hard-code") such defaults automatically during build time.

Many thanks,

Dimitri

[Stefano Babic]

Hi Dmitrios,

On 06.04.22 23:21, Dimitrios Bouras wrote:
> Hello,
>
> Currently it is possible to patch ("on the fly", e.g. by editing in
> place using "sed") file sw-description just before it's added into the
> SWU cpio archive because do_swupdate() copies it from $WORKDIR to $S (=
> ${WORKDIR}/${PN}).
>
> Unfortunately the same is not true for any other local file because
> swupdate_add_src_uri() (called by do_swupdate) copies everything else
> directly from SRC_URI, instead of from $WORKDIR (contrary to usual
> practice).

Well, you want to add some hacks...

>
> I understand there are advantages in doing things this way. I'm looking
> for an elegant "work-around" to patch local source files before they are
> copied from SRC_URI into $S (in preparation for the call to cpio for
> producing the .swu file).

A SWU is a "deployment" file: this means we have already built and we
need to pack all together in an update package (the SWU). With this
goal, sw-description is handled separately because it is often a
template. It is still possible to pick up file added to SRC_URI, but
they are thought as all artifacts : they are ready to be deployed.

If you need to change, you have to follow the general rule in OE: you
create a package and you deploy an artifact. So if you need to patch a
script, you create a package for this script, you add a task to patch
it, and then you add a do_deploy() to put the resulting file into
DEPLOYDIR. In fact, meta-swupdate expects that artifacts are in the
deploy directory. You add them your script to SWUPDATE_IMAGES, that's all.

>
> A bit of background for context: the files I'm interested in patching
> via "sed" are in fact scripts that have fall-back defaults saved in
> variables. It is desirable to generate (and "hard-code") such defaults
> automatically during build time.

See above.

Best regards,
Stefano Babic

>
> Many thanks,
> Dimitri
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/CAExTYs0mnoYaDhMrY7D260tT4EDE3nJjoVt7dXxzV03wdEHP5w%40mail.gmail.com
> <https://groups.google.com/d/msgid/swupdate/CAExTYs0mnoYaDhMrY7D260tT4EDE3nJjoVt7dXxzV03wdEHP5w%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[Dimitrios Bouras]

Thank you for the guidance Stefano, it is greatly appreciated!

Best,

Dimitri

[Dimitrios Bouras]

Hello again Stefano,

I have coded a workaround that a) does what I need, and b) short-circuits generation of the script artifact. What led me to this solution was realizing that the script *IS* the artifact that I need to deploy. An "image" in $DEPLOY_DIR_IMAGE can be any type of file I want, as long as naming conventions are satisfied. Below I have listed an example of the SWU package recipe with lots of comments to guide anyone that's interested.

Once again thank you for the feedback - it put me on the right track.

Wishing you a great weekend,

Dimitri

------------8<-------------8<-------------8<-------------8<-------------8<-------------8<------------

# Copyright (C) 2022 Dimitri Bouras <dimitrio...@gmail.com>
# Released under the MIT license (see COPYING.MIT for the terms)

DESCRIPTION = "Example for patching recipe-specific scripts at package build-time"
SECTION = ""
PV = "1.0"
PR = "r0"

# Note: sw-description is mandatory
SRC_URI_example-imx6ul = "file://sw-description \
    file://partition.sh \
    file://chgroup.sh \
    file://led-blink.sh \
"
inherit swupdate

LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

# IMAGE_DEPENDS: list of images that include a root filesystem,
# ensuring they are built before creating the swupdate package
IMAGE_DEPENDS = "os-image"

DEPENDS += "u-boot-script"

# SWUPDATE_IMAGES: list of images and binary files that will be part
# of the compound image - images must be in the DEPLOY directory.
# There's nothing stopping us from adding scripts as images: this is
# a workaround for patching scripts during package build time. Note
# the removal of the ".sh" extension for scripts added in this way.
SWUPDATE_IMAGES = " \
    os-image \
    boot.scr \
    u-boot \
    itbImage\
    chgroup \
    led-blink \
"

# Images may have multiple formats so define the format selected for the
# compound image. Note the ".sh" extension for scripts added as images.
SWUPDATE_IMAGES_FSTYPES[os-image] = ".cpio.gz.u-boot"
SWUPDATE_IMAGES_FSTYPES[u-boot] = ".imx-emmc"
SWUPDATE_IMAGES_FSTYPES[itbImage] = ".bin"
#
SWUPDATE_IMAGES_FSTYPES[chgroup] = ".sh"
SWUPDATE_IMAGES_FSTYPES[led-blink] = ".sh"

# Indicate that $MACHINE should not be appended to script files
# added to the SWU CPIO package via SWUPDATE_IMAGES.
# The $MACHINE suffix is needed, however, when generating patched
# scripts in $DEPLOY_DIR_IMAGE, since do_swuimage expects all its
# input image files in this way (see do_patchswupdate below).
SWUPDATE_IMAGES_NOAPPEND_MACHINE[chgroup] = "true"
SWUPDATE_IMAGES_NOAPPEND_MACHINE[led-blink] = "true"

# Patching files in $WORKDIR has meaning only for sw-description as
# this is the only file do_swupdate copies from $WORKDIR; it copies
# all other files directly from their SRC_URI location so patching
# in $WORKDIR has no effect.
# By adding the scripts we want to patch to SWUPDATE_IMAGES, we instruct
# do_swupdate to copy them from $DEPLOY_DIR_IMAGE. Then we save patched
# script output in ${DEPLOY_DIR_IMAGE}/<script name>-${MACHINE}.
# do_swupdate will then overwrite the non-patched scripts copied from
# SRC_URI with the patched ones from $DEPLOY_DIR_IMAGE. The $MACHINE
# suffix is removed as instructed by SWUPDATE_IMAGES_NOAPPEND_MACHINE.
do_patchswupdate() {
    # sw-decription is OK in $WORKDIR
    sed -i -e 's/__VERSION__/${PV}-${PR}/g' ${WORKDIR}/sw-description
    # the rest must go in $DEPLOY_DIR_IMAGE and be suffixed with $MACHINE
    sed -e 's/__UPDATEGROUP__/postmfg/g' ${WORKDIR}/chgroup.sh > ${DEPLOY_DIR_IMAGE}/chgroup.sh-${MACHINE}
    sed -e 's/__LEDNO__/2/g' ${WORKDIR}/led-blink.sh > ${DEPLOY_DIR_IMAGE}/led-blink.sh-${MACHINE}
}
addtask do_patchswupdate before do_swuimage after do_unpack

[Dimitrios Bouras]

Apologies for the mix-up; I attached an older (incorrect) version of the example recipe.

The correct one is below.

Best,

Dimitri

SWUPDATE_IMAGES_NOAPPEND_MACHINE[chgroup] = "true"
SWUPDATE_IMAGES_NOAPPEND_MACHINE[led-blink] = "true"

# Patching files in $WORKDIR has meaning only for sw-description as
# this is the only file do_swupdate copies from $WORKDIR; it copies
# all other files directly from their SRC_URI location so patching
# in $WORKDIR has no effect.
# By adding the scripts we want to patch to SWUPDATE_IMAGES, we instruct
# do_swupdate to copy them from $DEPLOY_DIR_IMAGE. Then we save patched

# script output in ${DEPLOY_DIR_IMAGE}/<script name>.

# do_swupdate will then overwrite the non-patched scripts copied from
# SRC_URI with the patched ones from $DEPLOY_DIR_IMAGE.

do_patchswupdate() {
    # sw-decription is OK in $WORKDIR
    sed -i -e 's/__VERSION__/${PV}-${PR}/g' ${WORKDIR}/sw-description
    # the rest must go in $DEPLOY_DIR_IMAGE

    sed -e 's/__UPDATEGROUP__/postmfg/g' ${WORKDIR}/chgroup.sh > ${DEPLOY_DIR_IMAGE}/chgroup.sh

    sed -e 's/__LEDNO__/2/g' ${WORKDIR}/led-blink.sh > ${DEPLOY_DIR_IMAGE}/led-blink.sh
}

addtask do_patchswupdate before do_swuimage after do_unpack

[Dimitrios Bouras]

One more adjustment to the example:

do_patchswupdate needs to be executed unconditionally (every time) before do_swuimage, not just after do_unpack, in order to rebuild the patched scripts anew.

The updated example source is attached.

Best,

Dimitri

# run do_patchswupdate unconditionally before do_swuimage, not just after do_unpack

addtask do_patchswupdate before do_swuimage

[Dimitrios Bouras]

When the same script source files are used by more than one recipe while patch contents are recipe-specific, bitbake must be instructed to run do_patchswupdate() unconditionally, ensuring correct patching during build time for every swupdate recipe. This is accomplished by setting the "nostamp" varflag for the task:

do_patchswupdate[nostamp] = "1"

The updated example recipe source is attached (apologies for the "noise" from repeated posts during the last few days).

do_patchswupdate[nostamp] = "1"