[meta-swupdate][PATCH 1/3] fix image generation when signing is enabled

[Christian Andersen]

When using signing openssl-native is required. Since pyro the sysroot
is deployed per recipe, so do_swuimage also requires
do_prepare_recipe_sysroot.

Signed-off-by: Christian Andersen <c.and...@kostal.com>
---
classes/swupdate.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index 28297ca..5a2d963 100644
--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass
@@ -199,5 +199,5 @@ python do_swuimage () {
COMPRESSIONTYPES = ""
PACKAGE_ARCH = "${MACHINE_ARCH}"

-addtask do_swuimage after do_unpack before do_install
+addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
addtask do_createlink after do_swuimage before do_install
--
2.7.4



KOSTAL Industrie Elektrik GmbH - Sitz Lüdenscheid, Registergericht Iserlohn HRB 3924 - USt-Id-Nr./Vat No.: DE 813742170
Postanschrift: An der Bellmerei 10, D-58513 Lüdenscheid * Telefon: +49 2351 16-0 * Telefax: +49 2351 16-2400
Werksanschrift: Lange Eck 11, D-58099 Hagen * Tel. +49 2331 8040-601 * Fax +49 2331 8040-602
Geschäftsführung: Dr.-Ing. Dipl.-Wirt.Ing. Manfred Gerhard, Dipl.-Ing. Marwin Kinzl, Dipl.-Oec. Andreas Kostal

[Christian Andersen]

Signed-off-by: Christian Andersen <c.and...@kostal.com>
---

classes/swupdate.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index 5a2d963..ef75d7f 100644

--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass
@@ -199,5 +199,5 @@ python do_swuimage () {
COMPRESSIONTYPES = ""
PACKAGE_ARCH = "${MACHINE_ARCH}"

-addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
-addtask do_createlink after do_swuimage before do_install
+addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_build
+addtask do_createlink after do_swuimage before do_build

[Stefano Babic]

On 09/09/2017 12:49, Christian Andersen wrote:
> When using signing openssl-native is required. Since pyro the sysroot
> is deployed per recipe, so do_swuimage also requires
> do_prepare_recipe_sysroot.
>
> Signed-off-by: Christian Andersen <c.and...@kostal.com>
> ---
> classes/swupdate.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index 28297ca..5a2d963 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -199,5 +199,5 @@ python do_swuimage () {
> COMPRESSIONTYPES = ""
> PACKAGE_ARCH = "${MACHINE_ARCH}"
>
> -addtask do_swuimage after do_unpack before do_install
> +addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
> addtask do_createlink after do_swuimage before do_install
>

Reviewed-by: Stefano Babic <sba...@denx.de>

Best regards,
Stefano Babic

--
=====================================================================
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sba...@denx.de
=====================================================================

[Stefano Babic]

On 09/09/2017 12:49, Christian Andersen wrote:

> Signed-off-by: Christian Andersen <c.and...@kostal.com>
> ---
> classes/swupdate.bbclass | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index 5a2d963..ef75d7f 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -199,5 +199,5 @@ python do_swuimage () {
> COMPRESSIONTYPES = ""
> PACKAGE_ARCH = "${MACHINE_ARCH}"
>
> -addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
> -addtask do_createlink after do_swuimage before do_install
> +addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_build
> +addtask do_createlink after do_swuimage before do_build
>

Sure you have already investigated, but the commit message and the
indication of the error remain cryptic to me.

Can you better explain which is the issue ? Is thi caused after applying
your first patch (it does not seem so..) ?

Thanks,

[Christian Andersen]

Hi,

On Thursday, 14 September 2017 09:48:53 UTC+2, Stefano Babic wrote:

On 09/09/2017 12:49, Christian Andersen wrote:
> Signed-off-by: Christian Andersen <c.and...@kostal.com>
> ---
>  classes/swupdate.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index 5a2d963..ef75d7f 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -199,5 +199,5 @@ python do_swuimage () {
>  COMPRESSIONTYPES = ""
>  PACKAGE_ARCH = "${MACHINE_ARCH}"
>  
> -addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
> -addtask do_createlink after do_swuimage before do_install
> +addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_build
> +addtask do_createlink after do_swuimage before do_build
>

Sure you have already investigated, but the commit message and the
indication of the error remain cryptic to me.

Can you better explain which is the issue ? Is thi caused after applying
your first patch (it does not seem so..) ?

Thanks,
Stefano Babic 

 

when deleting the tmp-dir of Yocto before a build, all unchanged recipes are

recreated from sstate. Unfortunately the swu images are not created in

this case, because the do_swuimage task is not run. After connecting the

do_swuimage task with do_build, the task is run even when recreating more or less

everything from sstate.

This is similar to the original image.bbclass where the task do_image_complete is

also connected to do_build. In swupdate.bbclass the task do_swuimage

or (more correct) do_createlink corresponds to do_image_complete in image.bbclass 

I have to admit that I did no further investigation why do_install is not working and

why do_build is okay.

The 3rd part of my patch series utilizes the sstate staging to copy the created swu

images to the final deploy directory (without caching the image itself). This is also

based on the current process used in pyro. Maybe both patches should be squashed.

Regards,

Christian 

[Stefano Babic]

Hi Christian,

On 14/09/2017 10:21, Christian Andersen wrote:
> Hi,
>
> On Thursday, 14 September 2017 09:48:53 UTC+2, Stefano Babic wrote:
>
> On 09/09/2017 12:49, Christian Andersen wrote:

> > Signed-off-by: Christian Andersen <c.and...@kostal.com <javascript:>>

Right.

> Unfortunately the swu images are not created in
> this case, because the do_swuimage task is not run. After connecting the
> do_swuimage task with do_build, the task is run even when recreating
> more or less
> everything from sstate.

Thanks, got it.

>
> This is similar to the original image.bbclass where the task
> do_image_complete is
> also connected to do_build. In swupdate.bbclass the task do_swuimage
> or (more correct) do_createlink corresponds to do_image_complete in
> image.bbclass

Right, thanks for explanation.

Waht about to add this explanation in the commit message ? This explains
the issues and how they are solved, and that the solution is already
used in yocto since pyro.

>
> I have to admit that I did no further investigation why do_install is
> not working and
> why do_build is okay.
>
> The 3rd part of my patch series utilizes the sstate staging to copy the
> created swu
> images to the final deploy directory (without caching the image itself).
> This is also
> based on the current process used in pyro. Maybe both patches should be
> squashed.

Agree. So please send V2, squashing 2/3 and 3/3, with description in the
commit message - thanks !

Best regards,
Stefano

[Andersen, Christian]

Hi Stefano,

> > I have to admit that I did no further investigation why do_install is
> > not working and why do_build is okay.
> >
> > The 3rd part of my patch series utilizes the sstate staging to copy
> > the created swu images to the final deploy directory (without caching
> > the image itself).
> > This is also
> > based on the current process used in pyro. Maybe both patches should
> > be squashed.
>
> Agree. So please send V2, squashing 2/3 and 3/3, with description in the commit
> message - thanks !

okay, thanks, I will send an update (today or early next week).

Regards
Christian

--
KOSTAL Industrie Elektrik GmbH
www.kostal-industrie-elektrik.com

[Christian Andersen]

Since morty YP uses the sstate also for the deploy directory. Using
this approach old images will be removed from deploy on a new build.
The same process is now used for updates images. As with normal images
the update images won't be cached in the sstate to reduce disk usage.

This change also resolves the issue, that update images are not
recreated after deleting the tmp directory and let bitbake regenerate
all unchanged recipes from sstate.

This is based on the original implementation in image.bbclass.

Signed-off-by: Christian Andersen <c.and...@kostal.com>
---

classes/swupdate.bbclass | 32 +++++++++++++++++++-------------
1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index 5a2d963..02db631 100644
--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass
@@ -64,15 +64,23 @@ def swupdate_getdepends(d):
depstr += " " + dep + ":do_build"
return depstr

+IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-swuimage"
+
do_swuimage[dirs] = "${TOPDIR}"
-do_swuimage[cleandirs] += "${S}"
+do_swuimage[cleandirs] += "${S} ${IMGDEPLOYDIR}"
do_swuimage[umask] = "022"
+SSTATETASKS += "do_swuimage"
+SSTATE_SKIP_CREATION_task-swuimage = '1'
+do_swuimage[sstate-inputdirs] = "${IMGDEPLOYDIR}"
+do_swuimage[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_swuimage[stamp-extra-info] = "${MACHINE}"

do_configure[noexec] = "1"
do_compile[noexec] = "1"
do_install[noexec] = "1"
+deltask do_populate_sysroot
do_package[noexec] = "1"
-do_package_qa[noexec] = "1"
+deltask do_package_qa
do_packagedata[noexec] = "1"
do_package_write_ipk[noexec] = "1"
do_package_write_deb[noexec] = "1"
@@ -83,14 +91,6 @@ python () {
d.appendVarFlag('do_swuimage', 'depends', deps)
}

-do_install () {
-}
-
-do_createlink () {
- cd ${DEPLOY_DIR_IMAGE}
- ln -sf ${IMAGE_NAME}.swu ${IMAGE_LINK_NAME}.swu
-}
-
python do_swuimage () {
import shutil

@@ -115,6 +115,7 @@ python do_swuimage () {
# If they are not there, additional file can be added
# by fetching from URLs
deploydir = d.getVar('DEPLOY_DIR_IMAGE', True)
+ imgdeploydir = d.getVar('IMGDEPLOYDIR', True)

for image in images:
fstypes = (d.getVarFlag("SWUPDATE_IMAGES_FSTYPES", image, True) or "").split()
@@ -192,12 +193,17 @@ python do_swuimage () {
else:
bb.fatal("Unrecognized SWUPDATE_SIGNING mechanism.");

- line = 'for i in ' + ' '.join(list_for_cpio) + '; do echo $i;done | cpio -ov -H crc >' + os.path.join(deploydir,d.getVar('IMAGE_NAME', True) + '.swu')
+ line = 'for i in ' + ' '.join(list_for_cpio) + '; do echo $i;done | cpio -ov -H crc >' + os.path.join(imgdeploydir,d.getVar('IMAGE_NAME', True) + '.swu')
os.system("cd " + s + ";" + line)
+
+ line = 'ln -sf ' + d.getVar('IMAGE_NAME', True) + '.swu ' + d.getVar('IMAGE_LINK_NAME', True) + '.swu'
+ os.system("cd " + imgdeploydir + "; " + line)

}

COMPRESSIONTYPES = ""
PACKAGE_ARCH = "${MACHINE_ARCH}"

-addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
-addtask do_createlink after do_swuimage before do_install

+INHIBIT_DEFAULT_DEPS = "1"
+EXCLUDE_FROM_WORLD = "1"
+

+addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_build

--
2.7.4

[Christian Andersen]

When using signing openssl-native is required. Since pyro the sysroot
is deployed per recipe, so do_swuimage also requires
do_prepare_recipe_sysroot.

Signed-off-by: Christian Andersen <c.and...@kostal.com>
---

classes/swupdate.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index 28297ca..5a2d963 100644
--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass

@@ -199,5 +199,5 @@ python do_swuimage () {

COMPRESSIONTYPES = ""
PACKAGE_ARCH = "${MACHINE_ARCH}"

-addtask do_swuimage after do_unpack before do_install
+addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install

addtask do_createlink after do_swuimage before do_install

[Stefano Babic]

On 20/09/2017 06:47, Christian Andersen wrote:
> When using signing openssl-native is required. Since pyro the sysroot
> is deployed per recipe, so do_swuimage also requires
> do_prepare_recipe_sysroot.
>
> Signed-off-by: Christian Andersen <c.and...@kostal.com>
> ---
> classes/swupdate.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index 28297ca..5a2d963 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -199,5 +199,5 @@ python do_swuimage () {
> COMPRESSIONTYPES = ""
> PACKAGE_ARCH = "${MACHINE_ARCH}"
>
> -addtask do_swuimage after do_unpack before do_install
> +addtask do_swuimage after do_unpack after do_prepare_recipe_sysroot before do_install
> addtask do_createlink after do_swuimage before do_install
>

Reviewed-by: Stefano Babic <sba...@denx.de>

Best regards,
Stefano Babic

[Stefano Babic]

Hi Christian,

Reviewed-by: Stefano Babic <sba...@denx.de>

Thanks for patch - I assume that this should applied (as 1/2) just to
-pyro, -master. Do you agree ?

[Andersen, Christian]

Hi Stefano,

> -----Ursprüngliche Nachricht-----
> Von: swup...@googlegroups.com [mailto:swup...@googlegroups.com] Im
> Gesendet: Mittwoch, 20. September 2017 09:52

>
> Hi Christian,
>
> On 20/09/2017 06:48, Christian Andersen wrote:
> > Since morty YP uses the sstate also for the deploy directory. Using
> > this approach old images will be removed from deploy on a new build.
> > The same process is now used for updates images. As with normal
> > images the update images won't be cached in the sstate to reduce disk usage.
> >
> > This change also resolves the issue, that update images are not
> > recreated after deleting the tmp directory and let bitbake
> > regenerate all unchanged recipes from sstate.
> >
> > This is based on the original implementation in image.bbclass.
> >
> > Signed-off-by: Christian Andersen <c.and...@kostal.com>
> >

> > (...)

> >
> >>
> Reviewed-by: Stefano Babic <sba...@denx.de>
>
> Thanks for patch - I assume that this should applied (as 1/2) just to -pyro, -master.
> Do you agree ?

I am currently using this on pyro.

Patch 1/2 is only needed for pyro and newer (because per-recipe-sysroots were implemented with pyro).
Patch 2/2 might be interesting for morty as well, because sstate for deploy dir was introduced in morty. But I did not test it on morty and I did not check if there were any changes in the sstate deployment implementation in images.bbclass between morty and pyro.

So for the moment: I would also say that pyro and master should be sufficient.

Regards
Christian

--
KOSTAL Industrie Elektrik GmbH
www.kostal-industrie-elektrik.com

[Stefano Babic]

Hi Christian,

On 20/09/2017 11:30, Andersen, Christian wrote:
> Hi Stefano,
>
>> -----Ursprüngliche Nachricht-----
>> Von: swup...@googlegroups.com [mailto:swup...@googlegroups.com] Im
>> Gesendet: Mittwoch, 20. September 2017 09:52
>>
>> Hi Christian,
>>
>> On 20/09/2017 06:48, Christian Andersen wrote:
>>> Since morty YP uses the sstate also for the deploy directory. Using
>>> this approach old images will be removed from deploy on a new build.
>>> The same process is now used for updates images. As with normal
>>> images the update images won't be cached in the sstate to reduce disk usage.
>>>
>>> This change also resolves the issue, that update images are not
>>> recreated after deleting the tmp directory and let bitbake
>>> regenerate all unchanged recipes from sstate.
>>>
>>> This is based on the original implementation in image.bbclass.
>>>
>>> Signed-off-by: Christian Andersen <c.and...@kostal.com>
>>>
>>> (...)
>>>
>>>>
>> Reviewed-by: Stefano Babic <sba...@denx.de>
>>
>> Thanks for patch - I assume that this should applied (as 1/2) just to -pyro, -master.
>> Do you agree ?
>
> I am currently using this on pyro.
>
> Patch 1/2 is only needed for pyro and newer (because per-recipe-sysroots were implemented with pyro).

Right.

> Patch 2/2 might be interesting for morty as well, because sstate for deploy dir was introduced in morty. But I did not test it on morty and I did not check if there were any changes in the sstate deployment implementation in images.bbclass between morty and pyro.

Agree. I wanted just to mention this on ML. I have also no time to test
it or fix it in -morty. Maybe someone will test and / or fix on -morty.

>
> So for the moment: I would also say that pyro and master should be sufficient.
>

Agree.

Best regards,
Stefano